Python-to-GPU Streaming with Arrow & GeoParquet

The fastest client-side WebGPU ingestion path is worthless if the server hands it bytes in the wrong shape. When a Python analytics backend reads a multi-million-point GeoParquet dataset and pushes it to a browser for rendering, every byte the server writes must land in a GPUBuffer whose memory layout the WGSL shader already expects — the same coordinate stride, the same scalar width, the same padding — or the client pays a repacking tax on the hot path that erases the benefit of GPU rendering entirely. This area covers the backend half of that contract: reading GeoParquet with pyarrow and GeoPandas, keeping coordinate columns in columnar Arrow buffers, downcasting float64 to float32 once on the server, arranging a structure-of-arrays layout that satisfies WGSL alignment, and framing the result so a chunked WebSocket stream feeds the client’s mapped buffer without a transform. It is the server-side companion to the binary-transport and zero-copy ingestion patterns established in Framework Integration & Backend Synchronization, and it depends throughout on the byte-layout rules derived in memory alignment for spatial data buffers.

The diagram fixes the governing principle: the layout decision lives on the server. A columnar Arrow table is already close to what the GPU wants — one dtype per contiguous run — so the transformation from GeoParquet to GPU-ready bytes is mostly a downcast and a concatenation, not a parse. Every section below tightens one link in that chain, from the pyarrow read through the wire frame to the buffer the client maps.

Prerequisites

1. A working WebGPU client that maps binary payloads into buffers using mappedAtCreation, as established in the ingestion patterns of the framework integration reference. This area produces the bytes that path consumes.
2. Familiarity with WGSL storage-buffer alignment — scalar f32 stride, vec2/vec3 padding, struct rounding — from memory alignment for spatial data buffers. The server layout must mirror those rules exactly.
3. A Python 3.10+ environment with pyarrow>=14, geopandas>=0.14, and numpy>=1.24. GeoParquet reads use pyarrow’s Parquet reader; coordinate extraction uses GeoPandas or direct WKB decoding.
4. A GeoParquet dataset that is already projected into the linear coordinate system the shader renders in (Web Mercator metres, typically). Streaming geographic degrees and projecting on the GPU wastes precision and shader cycles; project server-side, once.
5. A duplex transport — a WebSocket channel is assumed here — capable of carrying binary frames without base64 or JSON wrapping.

Layout and alignment reference

The server serializer and the client buffer descriptor are two encodings of one contract. This table is that contract: every row is a decision the Python side makes that the WGSL side depends on. It scrolls horizontally on narrow viewports.

Two rows deserve emphasis. The vec3 stride rule is the single most common byte-drift bug: WGSL rounds a vec3<f32> array element up to 16 bytes, so a server that ships tightly packed 12-byte triplets desynchronizes the whole array. Either pad to 16 on the server or, better, keep positions as vec2 and carry elevation in a parallel column. And endianness is a silent corruptor — it never raises an error, it just renders a scrambled map — so the wire format must be pinned to little-endian, which both numpy on mainstream hardware and the WebGPU memory model already assume.

Implementation walkthrough

The walkthrough builds the server serializer in stages: read GeoParquet columns, downcast and pack into a structure-of-arrays block, then frame it for the stream. The two deeper pages in this area carry the complete runnable programs; the steps here establish the shape each one fills in.

Step 1 — Read GeoParquet coordinate columns with pyarrow

Read only the columns the GPU needs. A GeoParquet file stores geometry as WKB by default, but projected point datasets are frequently materialized with plain x/y float columns precisely so downstream consumers can skip WKB decoding. Selecting columns at read time keeps the row group’s other attributes out of memory.

import pyarrow.parquet as pq
import numpy as np

def read_coordinate_columns(path: str) -> tuple[np.ndarray, np.ndarray, np.ndarray]:
    """Read x, y, and feature-id columns from a projected GeoParquet tile.

    Column projection (columns=...) means only these three runs are
    materialized; the WKB geometry blob and other attributes stay on disk.
    """
    table = pq.read_table(path, columns=["x", "y", "feature_id"])
    # zero_copy_only where the Arrow buffer is already contiguous and unmasked.
    x = table.column("x").to_numpy(zero_copy_only=False)   # f64
    y = table.column("y").to_numpy(zero_copy_only=False)   # f64
    ids = table.column("feature_id").to_numpy(zero_copy_only=False)  # int64/int32
    return x, y, ids

Reading columns rather than rows is the whole point of a columnar format: the x values sit in one contiguous Arrow buffer, the y values in another. That physical layout is already structure-of-arrays, which is exactly what a WGSL storage buffer wants — the downcast in the next step operates on contiguous runs with no gather. The complete streaming version, including row-group iteration for datasets that exceed memory, is the subject of streaming GeoParquet columns into WebGPU buffers.

Step 2 — Downcast f64 to f32 and pack structure-of-arrays

WebGPU storage buffers have no native double-precision type, so coordinates must become f32 before they cross the wire. Doing this once on the server halves the payload and removes per-frame client work.

def pack_soa(x: np.ndarray, y: np.ndarray, ids: np.ndarray) -> bytes:
    """Pack columns into a structure-of-arrays byte block: positions, then ids.

    Positions are interleaved x,y as vec2<f32> (stride 8 bytes) so the GPU
    reads array<vec2<f32>> with no gather. Ids follow as a separate u32 block.
    """
    n = x.shape[0]
    # Interleave into an (N,2) f32 buffer — one downcast, contiguous output.
    positions = np.empty((n, 2), dtype=np.float32)
    positions[:, 0] = x            # numpy casts f64 -> f32 on assignment
    positions[:, 1] = y
    positions = np.ascontiguousarray(positions)     # gap-free row-major

    ids32 = ids.astype(np.uint32, copy=False)       # match array<u32> binding

    # Contiguous SoA: [positions block][ids block]. Both 4-byte aligned.
    return positions.tobytes() + ids32.tobytes()

The interleaved (N, 2) float32 array is byte-identical to a WGSL array<vec2<f32>>: element i occupies bytes 8i .. 8i+8, x then y. Appending the uint32 id block keeps positions and attributes in separate contiguous runs, matching two distinct storage-buffer bindings on the client. Because degree-scale coordinates lose precision at f32, projected metre coordinates in a tile-local frame are the safe input; for centimetre-accurate data, subtract a tile origin server-side and ship offsets, restoring the high bits in the shader — the same mitigation the alignment reference documents.

Step 3 — Frame the block for chunked streaming

A dataset larger than one comfortable message is split into chunks, each carrying a small fixed-size header so the client can size its buffer before copying. The header travels in the same frame as its payload to keep framing atomic.

import struct

# Header: magic(u32) point_count(u32) positions_bytes(u32) ids_bytes(u32)
HEADER = struct.Struct("<IIII")
MAGIC = 0x47505541  # "GPUA" little-endian

def frame_chunk(positions_bytes: bytes, ids_bytes: bytes, count: int) -> bytes:
    """Prepend a fixed 16-byte little-endian header to one SoA chunk."""
    header = HEADER.pack(MAGIC, count, len(positions_bytes), len(ids_bytes))
    return header + positions_bytes + ids_bytes

The < in every format string pins little-endian, the one wire convention WebGPU mandates and the one an x86 or ARM server already produces — making it explicit documents intent and survives a big-endian build host. The client reads the 16-byte header, allocates a GPUBuffer sized to positions_bytes + ids_bytes rounded up to 4, and copies the remainder of the frame straight in. Sequencing many such chunks over a live channel, with backpressure and ordering, is detailed in the streaming reference; landing them zero-copy through Arrow’s buffer protocol is covered in zero-copy Arrow buffers to WebGPU storage.

Memory and performance implications

Downcasting on the server is the dominant win: an (N, 2) float64 coordinate array is 16 bytes per point, the f32 form 8 — a two-times reduction on the wire and in client VRAM, before any compression. For 10 million points that is 160 MiB versus 80 MiB, which is the difference between fitting and not fitting inside a default maxStorageBufferBindingSize. The id block adds 4 bytes per point; keeping it in a separate binding rather than interleaving into a per-point struct avoids the padding a mixed struct would force and keeps each binding’s stride a clean power of two.

The columnar read is close to free relative to a row-oriented parse. Because pyarrow hands back the x and y buffers as contiguous memory, the only copy is the interleave in Step 2, and even that can be elided when positions are already stored interleaved in the source file. The staging cost that remains is the tobytes() materialization; the zero-copy techniques in the deeper page remove it by handing Arrow’s underlying buffer to the socket directly.

Budgeting frames against the client’s buffer limits means treating $12N+16$ as the per-chunk size and choosing $N$ so a chunk stays under both the WebSocket message ceiling and the buffer binding limit. Tuning that chunk size against VRAM headroom is where this work meets the VRAM budget management across tile zoom levels guidance — a chunk that overflows the per-zoom budget forces an eviction the moment it lands.

Failure modes and diagnostics

• dtype mismatch (f64 read as f32). The server ships float64 coordinates but the client binds array<vec2<f32>>; the GPU reads each 8-byte double as two 4-byte floats, so every coordinate is garbage and the point cloud collapses to a smear near the origin. Detection: rendered points collapse onto a few impossible locations; the payload is exactly double the expected byte size. Fix: astype(np.float32) before tobytes(), and assert positions.dtype == np.float32 in the serializer.
• Endianness drift. A big-endian build host (or an explicit > in a struct format) writes reversed bytes; WebGPU reads little-endian, so coordinates and counts are byte-swapped. Detection: coordinates render mirrored or wildly out of range with no validation error. Fix: pin every struct format and numpy dtype to little-endian (<, '<f4'), and verify the header magic decodes on the client before trusting the payload.
• Padding drift on vec3. Positions shipped as tightly packed 12-byte vec3 triplets desync against WGSL’s 16-byte vec3 array stride, so every element after the first reads 4 bytes into the previous one. Detection: the first point is correct, all subsequent points progressively skewed. Fix: pad each vec3 to 16 bytes server-side, or carry elevation as a parallel f32 column and keep positions as vec2.
• WKB vs coordinate confusion. Reading the GeoParquet geometry column yields WKB blobs (a 1-byte order flag, a 4-byte type code, then coordinates), not raw floats; shipping those bytes as positions feeds the shader header bytes as coordinates. Detection: the first point per geometry is offset by a fixed 5-byte skew; counts do not match feature counts. Fix: decode WKB to x/y arrays server-side (via GeoPandas geometry.x/geometry.y or shapely), or read the materialized x/y columns directly as in Step 1.
• Unaligned payload size. A payload whose byte length is not a multiple of 4 makes the client round the GPUBuffer size up, leaving trailing bytes the shader may read as a phantom point. Detection: an extra stray point at the array tail; size and byteLength disagree by 1–3. Fix: ensure the positions and ids blocks are each a multiple of 4 bytes (they are by construction for f32/u32), and pass the exact summed length in the header.

Continue in this section

• Streaming GeoParquet Columns into WebGPU Buffers — the complete pyarrow-to-frame server program with row-group iteration, plus the client-side TypeScript mapping.
• Zero-Copy Arrow Buffers to WebGPU Storage — emit a payload from Arrow’s buffer protocol with no intermediate copy, mapped straight into a storage buffer.

Related

• Framework Integration & Backend Synchronization — the binary-transport and zero-copy ingestion contract this area serves.
• Memory Alignment for Spatial Data Buffers — the WGSL byte-layout rules the server serializer must mirror.
• VRAM Budget Management Across Tile Zoom Levels — sizing streamed chunks against the client’s memory budget.
• Reducing GPU Memory Fragmentation During Spatial Aggregation — the pooled staging buffers these streamed batches land in.

Up: Framework Integration & Backend Synchronization