Chunking Arrow Record Batches for Progressive GPU Upload

An Arrow response that arrives as one record batch cannot paint until its last byte lands; the same response split into batches can paint after the first. That is the whole argument for chunking, and the interesting part is that the relationship between batch size and time-to-first-geometry has a floor rather than an asymptote — past a certain point, smaller batches make things worse because per-upload overhead starts to dominate. This page covers where that floor sits, how the client consumes a stream of batches into one pre-allocated buffer, and the row count that has to travel ahead of the data. It is one stage of Python to GPU streaming with Arrow and GeoParquet.

Time to first geometry against batch size A bar chart in milliseconds of the delay before any geometry appears, for a 4.8 million row response under four batching choices. A single batch waits 1420 milliseconds for the whole payload. Batches of 500 000 rows wait 180 milliseconds. Batches of 100 000 rows wait 52 milliseconds, the minimum. Batches of 10 000 rows wait 96 milliseconds, because 480 separate uploads cost more in fixed overhead than the smaller wait saves. TIME TO FIRST GEOMETRY · ms One batch 1420 ms 500 K rows 180 ms 100 K rows 52 ms — the floor 10 K rows 96 ms 0 300 600 900 1200 1500 ms waits for everything workable best The turn is a byte threshold, not a row count — it moves with how wide the rows are.
The curve has a floor rather than an asymptote, and the last row is the one people find surprising: past the turn, smaller batches make first paint later, not sooner.

Runnable reference implementation

The buffer is allocated once for the full row count and filled progressively. Nothing is ever reallocated, because a reallocation invalidates every bind group already built against the buffer.

typescript
interface StreamTarget {
  buffer: GPUBuffer;         // allocated for totalRows * stride
  stride: number;            // bytes per row
  written: number;           // rows written so far — also the instance count
}

async function streamBatches(
  device: GPUDevice,
  response: Response,
  totalRows: number,         // from a header — see the note below
  stride: number,
): Promise<StreamTarget> {
  const target: StreamTarget = {
    buffer: device.createBuffer({
      label: "streamed-points",
      size: totalRows * stride,
      usage: GPUBufferUsage.VERTEX | GPUBufferUsage.COPY_DST,
    }),
    stride,
    written: 0,
  };

  for await (const batch of readArrowBatches(response.body!)) {
    const column = batch.getChildAt(0)!;             // the coordinate column
    const view = new Float32Array(
      column.data.buffers[1].buffer,
      column.data.buffers[1].byteOffset,
      batch.numRows * (stride / Float32Array.BYTES_PER_ELEMENT),
    );
    device.queue.writeBuffer(target.buffer, target.written * stride, view);
    target.written += batch.numRows;                 // the draw grows with it
  }
  return target;
}

The draw call reads written as its instance count, so the map paints exactly the rows that have arrived and nothing else. No branch, no placeholder geometry, no special case for the partially-loaded state — the count is simply smaller for a while.

Parameter reference

Value Setting here Guidance
Rows per batch ~100 000 The floor for a narrow row. Wider rows reach it at fewer rows, since the knob is really bytes per batch.
Bytes per batch ~1–2 MiB The more portable way to state the same thing; tune this rather than the row count.
Total row count in a header The client must know it before the first batch to size the buffer.
Buffer usage `VERTEX COPY_DST`
Reallocation never Invalidates every bind group; allocate to the total up front.
One buffer, filled as the batches land A single strip of twelve slots represents a buffer allocated once for the full row count. Five slots are shaded as written, marking the batches that have arrived and been uploaded into their own byte ranges. The remaining seven are allocated but not yet written. The draw call reads its instance count from the written prefix, so the map shows exactly the rows that have landed and no placeholder geometry is needed for the rest. BATCH SLOT 0 1 2 3 4 5 6 7 8 9 10 11 12 b1 b2 b3 b4 b5 allocated, not yet written Buffer sized to total rows the instance count follows the written prefix — no special case for partial data This is why the total row count has to arrive before the first batch does.
Allocating for the total up front is what makes the stream simple. A buffer that grows would invalidate every bind group referencing it, which is the one thing a progressive upload cannot afford mid-stream.

Where the floor comes from

Each writeBuffer has a fixed cost — validation, a staging allocation inside the implementation, and a queue operation — that is independent of how many bytes it moves. At a megabyte per call that cost is noise; at ten kilobytes it is most of the work.

The crossover is what produces the U-shaped curve. Halving the batch size halves the time until the first batch arrives, which is a real win, and doubles the number of uploads, which is a real cost. While the first effect dominates, smaller is better; once the second does, smaller is worse. For a typical narrow row the turn is somewhere around one to two megabytes per batch, which at eight bytes per row is a few hundred thousand rows.

That framing also explains why the optimum moves with row width rather than row count. A row carrying twelve attributes reaches the same byte threshold at a tenth of the rows, so a batch size expressed in rows and tuned on one dataset will be badly wrong on another. Expressing it in bytes and deriving the row count from the stride is the portable version.

The last consideration is the network. Batches arriving over a slow connection are already spread out in time, so the per-upload overhead is hidden by the wait — which means the optimum on a fast local connection is not the optimum in production. Tuning against a throttled connection is closer to the truth.

What to do when the row count is unknown

Sizing the buffer up front needs a total, and there are pipelines where the server genuinely cannot supply one — a query whose result set is computed as it streams, or a live feed with no end. Three responses work, in increasing order of complexity.

The simplest is to allocate to a cap. Pick a maximum the application is willing to render — a few million rows — allocate for it, and treat overflow as a truncation with a visible warning. That is honest, costs one allocation, and for an interactive map is usually correct: nobody is reading four million individually rendered points anyway.

The second is a chain of buffers. When one fills, allocate another and draw both. The bind groups for the first stay valid, which is the property that matters, and the draw becomes two calls instead of one. It handles unbounded input at the cost of a growing draw list, and it is the right answer for a live feed.

The third is a compaction pass. When the chain gets long, run a compute pass that copies the live rows into one new buffer and retire the old ones. That is real work and real complexity, and it is only worth it for a long-running session where the chain would otherwise grow without bound.

What does not work is reallocating and copying in place, because the moment the buffer identity changes every bind group referencing it is invalid — and rebuilding those mid-stream is exactly the stall the progressive upload was designed to avoid.

Ordering, and whether it matters

Arrow record batches arrive in the order the server wrote them, and over HTTP that order is preserved. That gives the client one property worth knowing about: the rows in the buffer are in the same order as the rows in the source table.

For most rendering that is irrelevant — points are drawn as a set. It becomes relevant in three cases. If the source was sorted spatially, as the Morton-ordering work recommends, the buffer inherits that locality and every later pass reads it coalesced. If the client needs to correlate a rendered feature with a row in a table shown alongside the map, the index into the buffer is the index into the table, which makes picking trivial. And if a second column arrives in a separate request — geometry from one endpoint, attributes from another — the two only line up if both preserved the order.

That last case is worth guarding rather than assuming. A server that parallelises its query across workers can return rows in completion order rather than source order unless it is explicitly told not to, and the symptom is attributes attached to the wrong features, which looks like a data problem and is a transport one.

Failure modes

  • Nothing renders until the whole response arrives. The batches are being collected into an array and uploaded at the end. The upload has to happen per batch.
  • A RangeError constructing the typed-array view. The Arrow buffer’s byte offset is not aligned to the element size, which happens with a hand-rolled serialiser that does not pad. Use the standard IPC writer.
  • The draw call renders garbage past the loaded rows. The instance count was set to the total rather than to written.
  • Frame hitches during the stream. Batches are too large, or several arrive in one frame. Bound the uploads per frame exactly as a tile pipeline does.
  • The buffer is reallocated mid-stream. The total row count was not known up front, so the client guessed and had to grow. Send it in a header.
Which knob to turn, and what it moves A table of four tuning choices with what each one improves and what it costs. Smaller batches improve time to first paint and cost more fixed per-upload overhead. Larger batches improve total throughput and cost a later first paint. Bounding uploads per frame improves frame smoothness and costs a slightly later completion. Sizing batches by bytes rather than rows improves portability across datasets and costs nothing at all. BATCH TUNING · IMPROVES · COSTS Improves Costs Smaller batches first paint per-upload overhead Larger batches total throughput later first paint Bound per frame frame smoothness slightly later finish Size by bytes portability nothing Tune against a throttled connection — a fast local link hides the per-upload cost entirely.
The last row is free and the one most often skipped. A batch size tuned in rows on a narrow dataset is badly wrong on a wide one, and the fix is one division by the stride.

Backend / Python interop note

pyarrow writes batched IPC streams directly, and the batch size is a parameter of the write rather than something the client can influence — which makes it a server-side tuning decision.

python
import pyarrow as pa
import pyarrow.ipc as ipc

def stream_batches(table: pa.Table, target_bytes: int = 1_500_000):
    """Yield IPC-framed batches sized by bytes rather than by rows."""
    row_bytes = table.nbytes / max(table.num_rows, 1)
    rows_per_batch = max(int(target_bytes / row_bytes), 1)
    sink = pa.BufferOutputStream()
    with ipc.new_stream(sink, table.schema) as writer:
        for batch in table.to_batches(max_chunksize=rows_per_batch):
            writer.write_batch(batch)
    return sink.getvalue()

Two things belong in the response alongside the data. The total row count, as a header, because the client sizes its buffer before the first batch arrives. And the schema version, so a client can refuse a layout it does not recognise rather than reading the wrong columns — the same versioning discipline the zero-copy path depends on.

Sizing by bytes rather than by rows, as above, is what makes one endpoint serve narrow point data and wide attribute tables without separate tuning.