Streaming GeoParquet Columns into WebGPU Buffers

The concrete problem is delivering a GeoParquet dataset too large to hold in one message — tens of millions of projected points — to a WebGPU client as an ordered sequence of binary frames, each of which the client copies into a GPUBuffer with no repacking. A single read_table followed by one giant tobytes() would materialize the whole dataset in server memory and block the socket until the last byte is packed. The streaming answer iterates the file’s row groups, downcasts and packs each group into a structure-of-arrays chunk, prepends a fixed header, and writes the frame — so memory stays bounded to one row group and the client can begin mapping buffers before the file is fully read. This is the runnable server behind the framing steps sketched in Python-to-GPU streaming with Arrow & GeoParquet; the byte layout it emits follows memory alignment for spatial data buffers.

Runnable reference implementation

The server below opens a GeoParquet file, streams it row group by row group over a WebSocket, and emits one framed structure-of-arrays chunk per group. Each frame is header + positions + ids, all little-endian, positions as interleaved vec2<f32> and ids as u32. The transport uses the websockets library, but the framing is transport-agnostic — the same frame_chunk bytes work over any duplex channel.

# stream_server.py — stream GeoParquet row groups as GPU-ready binary frames.
import asyncio
import struct

import numpy as np
import pyarrow.parquet as pq
import websockets

# Fixed 16-byte frame header, little-endian to match the WebGPU memory model.
# magic | point_count | positions_byte_len | ids_byte_len
HEADER = struct.Struct("<IIII")
MAGIC = 0x47505541  # "GPUA"

# A trailing zero-count frame signals end-of-stream to the client.
END_FRAME = HEADER.pack(MAGIC, 0, 0, 0)


def pack_soa(x: np.ndarray, y: np.ndarray, ids: np.ndarray) -> tuple[bytes, bytes]:
    """Downcast and interleave one row group into (positions_bytes, ids_bytes).

    Positions: (N,2) float32, x then y — byte-identical to array<vec2<f32>>.
    Ids:       (N,)  uint32     — a separate contiguous storage-buffer block.
    """
    n = x.shape[0]
    positions = np.empty((n, 2), dtype=np.float32)
    positions[:, 0] = x                      # f64 -> f32 on assignment
    positions[:, 1] = y
    positions = np.ascontiguousarray(positions)   # gap-free row-major
    ids32 = np.ascontiguousarray(ids.astype(np.uint32, copy=False))
    return positions.tobytes(), ids32.tobytes()


def frame_chunk(positions_bytes: bytes, ids_bytes: bytes, count: int) -> bytes:
    """Prepend the fixed header to one packed chunk."""
    header = HEADER.pack(MAGIC, count, len(positions_bytes), len(ids_bytes))
    return header + positions_bytes + ids_bytes


async def stream_geoparquet(websocket, path: str) -> None:
    """Send one framed chunk per row group, then an end-of-stream frame."""
    pf = pq.ParquetFile(path)
    for rg in range(pf.num_row_groups):
        # Read only the coordinate + id columns of this single row group,
        # so peak memory is bounded to one group, not the whole file.
        table = pf.read_row_group(rg, columns=["x", "y", "feature_id"])
        x = table.column("x").to_numpy(zero_copy_only=False)
        y = table.column("y").to_numpy(zero_copy_only=False)
        ids = table.column("feature_id").to_numpy(zero_copy_only=False)

        if x.shape[0] == 0:
            continue                          # skip empty groups, emit no frame

        pos_bytes, id_bytes = pack_soa(x, y, ids)
        frame = frame_chunk(pos_bytes, id_bytes, x.shape[0])

        # Guard the 4-byte alignment the client relies on for buffer sizing.
        assert (len(pos_bytes) + len(id_bytes)) % 4 == 0, "payload not 4-aligned"

        await websocket.send(frame)           # one binary WebSocket message
        # Yield so backpressure from a slow client is respected, not buffered.
        await asyncio.sleep(0)

    await websocket.send(END_FRAME)


async def handler(websocket):
    # In production, derive the path from an authenticated, validated request.
    await stream_geoparquet(websocket, "points_z12.parquet")


async def main() -> None:
    # max_size=None: frames can exceed the default 1 MiB text ceiling.
    async with websockets.serve(handler, "0.0.0.0", 8765, max_size=None):
        await asyncio.Future()                # run until cancelled


if __name__ == "__main__":
    asyncio.run(main())

The load-bearing decision is read_row_group inside the loop rather than read_table before it: peak server memory is one row group’s coordinate columns, so the process streams a 40-million-point file in the footprint of its largest group. Column projection (columns=[...]) keeps every other attribute — including the WKB geometry blob a GeoParquet file usually carries — off the read path entirely. The await asyncio.sleep(0) after each send hands control back to the event loop so a slow consumer applies real backpressure instead of the server buffering unbounded frames in memory.

Client-side mapping

The client reads each frame’s 16-byte header, sizes a GPUBuffer, and copies the payload straight into mapped memory. This TypeScript sketch is the counterpart to the server frame; the position block binds as array<vec2<f32>> and the id block as array<u32> with no transform.

// Decode one framed chunk into two GPU buffers, byte-for-byte.
function ingestFrame(device: GPUDevice, frame: ArrayBuffer) {
  const view = new DataView(frame);
  const magic = view.getUint32(0, true);           // little-endian
  if (magic !== 0x47505541) throw new Error("bad frame magic");
  const count = view.getUint32(4, true);
  if (count === 0) return null;                     // end-of-stream frame
  const posLen = view.getUint32(8, true);
  const idLen = view.getUint32(12, true);

  const positions = device.createBuffer({
    size: posLen,                                   // already 4-aligned (f32)
    usage: GPUBufferUsage.STORAGE,
    mappedAtCreation: true,
  });
  new Uint8Array(positions.getMappedRange())
    .set(new Uint8Array(frame, 16, posLen));        // single copy, no parse
  positions.unmap();

  const ids = device.createBuffer({
    size: idLen,
    usage: GPUBufferUsage.STORAGE,
    mappedAtCreation: true,
  });
  new Uint8Array(ids.getMappedRange())
    .set(new Uint8Array(frame, 16 + posLen, idLen));
  ids.unmap();

  return { positions, ids, count };
}

The 16 and 16 + posLen offsets are the header size and the position-block length from the same header — reading them from the frame rather than assuming them is what keeps the client resilient to a chunk-size change on the server.

Parameter reference

Every tunable in the server, with guidance for streamed spatial workloads. Table scrolls horizontally on narrow viewports.

Failure modes

• Whole file loaded into memory. Calling read_table(path) instead of read_row_group in the loop defeats streaming: the server materializes every point before the first frame ships, spiking RSS and delaying first paint. Detection: server memory scales with file size; the client sees nothing until a long pause ends. Fix: iterate pf.num_row_groups and read one group per frame, as shown.
• Frame larger than the transport ceiling. A row group that packs past the WebSocket max_size is rejected and the connection drops mid-stream. Detection: the socket closes with a 1009 (message too big) after a specific large group. Fix: set max_size=None and cap row-group size at write time so no single frame exceeds the client’s buffer limits.
• Header/payload desync. Sending the header and payload as two separate messages (or mis-summing the lengths) makes the client read positions from the wrong offset. Detection: coordinates shift by a fixed byte count; the id block reads coordinate bytes. Fix: concatenate header and payload into one frame and let the client derive all offsets from the header fields, never from constants.
• int64 ids silently truncated. GeoParquet feature ids are often int64; astype(np.uint32) wraps values above 2^32 without warning. Detection: distinct features collide on the same id after some threshold. Fix: range-check ids before narrowing, or carry ids as two u32 columns (high/low) if the full 64-bit range is genuinely used.

Backend / Python interop note

The row-group loop is also the natural place to enforce the per-tile point budget that keeps the client’s pooled staging buffers stable. If a source file was written with oversized row groups, rewrite it with a bounded group size (pq.write_table(..., row_group_size=1_000_000)) rather than splitting frames on the fly — the write-time size is what fixes both memory footprint and frame granularity. For datasets that are dense enough that even one group overflows the client’s VRAM budget across zoom levels, pre-tile spatially so each file covers one tile, and stream files in viewport priority order.

Related

• Python-to-GPU Streaming with Arrow & GeoParquet — the layout contract and framing overview this server implements.
• Zero-Copy Arrow Buffers to WebGPU Storage — remove the tobytes() copy by shipping Arrow’s buffer directly.
• Memory Alignment for Spatial Data Buffers — the stride and padding rules the packed frames obey.

Up: Python-to-GPU Streaming with Arrow & GeoParquet