Streaming GeoParquet Columns into WebGPU Buffers

The concrete problem is delivering a GeoParquet dataset too large to hold in one message — tens of millions of projected points — to a WebGPU client as an ordered sequence of binary frames, each of which the client copies into a GPUBuffer with no repacking. A single read_table followed by one giant tobytes() would materialize the whole dataset in server memory and block the socket until the last byte is packed. The streaming answer iterates the file’s row groups, downcasts and packs each group into a structure-of-arrays chunk, prepends a fixed header, and writes the frame — so memory stays bounded to one row group and the client can begin mapping buffers before the file is fully read. This is the runnable server behind the framing steps sketched in Python-to-GPU streaming with Arrow & GeoParquet; the byte layout it emits follows memory alignment for spatial data buffers.

Runnable reference implementation

The server below opens a GeoParquet file, streams it row group by row group over a WebSocket, and emits one framed structure-of-arrays chunk per group. Each frame is header + positions + ids, all little-endian, positions as interleaved vec2<f32> and ids as u32. The transport uses the websockets library, but the framing is transport-agnostic — the same frame_chunk bytes work over any duplex channel.

python
# stream_server.py — stream GeoParquet row groups as GPU-ready binary frames.
import asyncio
import struct

import numpy as np
import pyarrow.parquet as pq
import websockets

# Fixed 16-byte frame header, little-endian to match the WebGPU memory model.
# magic | point_count | positions_byte_len | ids_byte_len
HEADER = struct.Struct("<IIII")
MAGIC = 0x47505541  # "GPUA"

# A trailing zero-count frame signals end-of-stream to the client.
END_FRAME = HEADER.pack(MAGIC, 0, 0, 0)


def pack_soa(x: np.ndarray, y: np.ndarray, ids: np.ndarray) -> tuple[bytes, bytes]:
    """Downcast and interleave one row group into (positions_bytes, ids_bytes).

    Positions: (N,2) float32, x then y — byte-identical to array<vec2<f32>>.
    Ids:       (N,)  uint32     — a separate contiguous storage-buffer block.
    """
    n = x.shape[0]
    positions = np.empty((n, 2), dtype=np.float32)
    positions[:, 0] = x                      # f64 -> f32 on assignment
    positions[:, 1] = y
    positions = np.ascontiguousarray(positions)   # gap-free row-major
    ids32 = np.ascontiguousarray(ids.astype(np.uint32, copy=False))
    return positions.tobytes(), ids32.tobytes()


def frame_chunk(positions_bytes: bytes, ids_bytes: bytes, count: int) -> bytes:
    """Prepend the fixed header to one packed chunk."""
    header = HEADER.pack(MAGIC, count, len(positions_bytes), len(ids_bytes))
    return header + positions_bytes + ids_bytes


async def stream_geoparquet(websocket, path: str) -> None:
    """Send one framed chunk per row group, then an end-of-stream frame."""
    pf = pq.ParquetFile(path)
    for rg in range(pf.num_row_groups):
        # Read only the coordinate + id columns of this single row group,
        # so peak memory is bounded to one group, not the whole file.
        table = pf.read_row_group(rg, columns=["x", "y", "feature_id"])
        x = table.column("x").to_numpy(zero_copy_only=False)
        y = table.column("y").to_numpy(zero_copy_only=False)
        ids = table.column("feature_id").to_numpy(zero_copy_only=False)

        if x.shape[0] == 0:
            continue                          # skip empty groups, emit no frame

        pos_bytes, id_bytes = pack_soa(x, y, ids)
        frame = frame_chunk(pos_bytes, id_bytes, x.shape[0])

        # Guard the 4-byte alignment the client relies on for buffer sizing.
        assert (len(pos_bytes) + len(id_bytes)) % 4 == 0, "payload not 4-aligned"

        await websocket.send(frame)           # one binary WebSocket message
        # Yield so backpressure from a slow client is respected, not buffered.
        await asyncio.sleep(0)

    await websocket.send(END_FRAME)


async def handler(websocket):
    # In production, derive the path from an authenticated, validated request.
    await stream_geoparquet(websocket, "points_z12.parquet")


async def main() -> None:
    # max_size=None: frames can exceed the default 1 MiB text ceiling.
    async with websockets.serve(handler, "0.0.0.0", 8765, max_size=None):
        await asyncio.Future()                # run until cancelled


if __name__ == "__main__":
    asyncio.run(main())

The load-bearing decision is read_row_group inside the loop rather than read_table before it: peak server memory is one row group’s coordinate columns, so the process streams a 40-million-point file in the footprint of its largest group. Column projection (columns=[...]) keeps every other attribute — including the WKB geometry blob a GeoParquet file usually carries — off the read path entirely. The await asyncio.sleep(0) after each send hands control back to the event loop so a slow consumer applies real backpressure instead of the server buffering unbounded frames in memory.

Client-side mapping

Streaming record batches so the first tile draws early Two lanes compare a single large response with a chunked stream. In the single-response arrangement the browser waits for the whole 48 mebibyte payload, then uploads it, then draws — nothing appears until every byte has arrived. In the chunked arrangement each Arrow record batch is uploaded into its slice of a pre-allocated buffer as it arrives, and the draw call’s instance count is raised after each batch, so the map paints progressively and the first geometry appears after the first batch rather than after the last. ONE RESPONSE vs STREAMED BATCHES Single response Streamed Await 48 MiB nothing is drawn yet Upload everything one writeBuffer First paint after the last byte Batch 1 arrives upload into slice 0 First paint instance count raised Batches 2..n each extends the draw Batch size is a latency knob: smaller batches paint sooner and cost more writeBuffer calls.
The buffer is allocated once for the full row count and filled progressively, so no reallocation happens mid-stream. Raising the instance count after each batch is what turns the upload into a progressive render rather than a loading screen.

The client reads each frame’s 16-byte header, sizes a GPUBuffer, and copies the payload straight into mapped memory. This TypeScript sketch is the counterpart to the server frame; the position block binds as array<vec2<f32>> and the id block as array<u32> with no transform.

typescript
// Decode one framed chunk into two GPU buffers, byte-for-byte.
function ingestFrame(device: GPUDevice, frame: ArrayBuffer) {
  const view = new DataView(frame);
  const magic = view.getUint32(0, true);           // little-endian
  if (magic !== 0x47505541) throw new Error("bad frame magic");
  const count = view.getUint32(4, true);
  if (count === 0) return null;                     // end-of-stream frame
  const posLen = view.getUint32(8, true);
  const idLen = view.getUint32(12, true);

  const positions = device.createBuffer({
    size: posLen,                                   // already 4-aligned (f32)
    usage: GPUBufferUsage.STORAGE,
    mappedAtCreation: true,
  });
  new Uint8Array(positions.getMappedRange())
    .set(new Uint8Array(frame, 16, posLen));        // single copy, no parse
  positions.unmap();

  const ids = device.createBuffer({
    size: idLen,
    usage: GPUBufferUsage.STORAGE,
    mappedAtCreation: true,
  });
  new Uint8Array(ids.getMappedRange())
    .set(new Uint8Array(frame, 16 + posLen, idLen));
  ids.unmap();

  return { positions, ids, count };
}

The 16 and 16 + posLen offsets are the header size and the position-block length from the same header — reading them from the frame rather than assuming them is what keeps the client resilient to a chunk-size change on the server.

Parameter reference

Time to first paint against record-batch size A bar chart in milliseconds of time to first geometry on screen for a 4.8 million row response, plotted against the number of rows per Arrow record batch. With one batch of 4.8 million rows, first paint waits 1420 milliseconds for the whole payload. At 500000 rows per batch it falls to 180 milliseconds. At 100000 rows it falls to 52 milliseconds. At 10000 rows it rises again to 96 milliseconds, because the per-batch overhead of 480 separate writeBuffer calls now dominates. TIME TO FIRST GEOMETRY · 4.8 M ROWS · ms 1 batch 1420 ms 500 K rows 180 ms 100 K rows 52 ms — best 10 K rows 96 ms 0 300 600 900 1200 1500 ms waits for everything workable best The optimum scales with row width, not row count — wider rows reach the floor at fewer rows per batch.
The curve has a floor, not an asymptote. Shrinking batches buys latency until the fixed cost of each upload takes over, and the minimum for this payload sits around a hundred thousand rows.

Every tunable in the server, with guidance for streamed spatial workloads. Table scrolls horizontally on narrow viewports.

Parameter Typical value Guidance
Row-group size (write time) 500 K–2 M rows Set when the GeoParquet is written; it fixes the frame granularity. Larger groups amortize header overhead; smaller groups lower peak memory and first-frame latency.
columns ["x", "y", "feature_id"] Project to exactly the GPU columns. Never read geometry (WKB) unless decoding it — it doubles the read.
HEADER format "<IIII" Little-endian, 16 bytes. The < is mandatory: WebGPU reads little-endian regardless of host byte order.
MAGIC 0x47505541 Sentinel the client checks before trusting a frame; catches desync and stray text frames.
max_size None The websockets default 1 MiB ceiling rejects large chunks; disable it or keep chunks under it.
positions dtype np.float32 Mandatory downcast — WebGPU storage buffers have no f64. Assert it before tobytes().
ids dtype np.uint32 Matches array<u32>; int64 ids must be narrowed (and range-checked) first.
End-of-stream signal zero-count frame Lets the client stop cleanly without relying on socket close, which can race in-flight frames.

Failure modes

One pre-allocated buffer filled batch by batch A single strip of twelve slots represents a buffer allocated once for the full row count. Four slots are shaded as written, marking the first four record batches that have arrived and been uploaded into their own byte range. The remaining eight are shown empty, allocated but not yet written. The draw call’s instance count tracks the written prefix, so the map renders exactly the rows that have arrived and nothing else. BATCH SLOT 0 1 2 3 4 5 6 7 8 9 10 11 12 b1 b2 b3 b4 allocated, not yet written Buffer allocated once the draw call’s instance count follows the written prefix If the total row count is unknown, allocate to a generous cap and treat overflow as a second buffer.
Allocating for the full row count up front is what lets the buffer never be reallocated mid-stream. A buffer that grows would invalidate every bind group referencing it, which is the one thing a progressive upload cannot afford.
  • Whole file loaded into memory. Calling read_table(path) instead of read_row_group in the loop defeats streaming: the server materializes every point before the first frame ships, spiking RSS and delaying first paint. Detection: server memory scales with file size; the client sees nothing until a long pause ends. Fix: iterate pf.num_row_groups and read one group per frame, as shown.
  • Frame larger than the transport ceiling. A row group that packs past the WebSocket max_size is rejected and the connection drops mid-stream. Detection: the socket closes with a 1009 (message too big) after a specific large group. Fix: set max_size=None and cap row-group size at write time so no single frame exceeds the client’s buffer limits.
  • Header/payload desync. Sending the header and payload as two separate messages (or mis-summing the lengths) makes the client read positions from the wrong offset. Detection: coordinates shift by a fixed byte count; the id block reads coordinate bytes. Fix: concatenate header and payload into one frame and let the client derive all offsets from the header fields, never from constants.
  • int64 ids silently truncated. GeoParquet feature ids are often int64; astype(np.uint32) wraps values above 2^32 without warning. Detection: distinct features collide on the same id after some threshold. Fix: range-check ids before narrowing, or carry ids as two u32 columns (high/low) if the full 64-bit range is genuinely used.

Backend / Python interop note

The row-group loop is also the natural place to enforce the per-tile point budget that keeps the client’s pooled staging buffers stable. If a source file was written with oversized row groups, rewrite it with a bounded group size (pq.write_table(..., row_group_size=1_000_000)) rather than splitting frames on the fly — the write-time size is what fixes both memory footprint and frame granularity. For datasets that are dense enough that even one group overflows the client’s VRAM budget across zoom levels, pre-tile spatially so each file covers one tile, and stream files in viewport priority order.

One operational note. Because the buffer is allocated for the full row count before the first batch arrives, the row count has to be known up front — which means the endpoint should send it in a header or in the schema metadata rather than making the client infer it. Without it the client either allocates to a guessed cap and wastes VRAM, or discovers mid-stream that it must reallocate, which invalidates every bind group already built against the buffer and drops whatever has been drawn so far.

Up: Python-to-GPU Streaming with Arrow & GeoParquet