Relative-to-Eye Encoding for f32 Coordinate Precision

The problem is one subtraction in the wrong place. A vertex at Web Mercator x = 16 700 000 and a camera at x = 16 700 240 are 240 metres apart, but in f32 both round to the same 2-metre grid, so their difference is quantised before it is ever computed. Doing the subtraction on the host in f64 — where JavaScript numbers already are — and uploading only the small residual recovers every bit of that precision at no run-time cost. This page is the complete encoding: what to subtract, where the origin lives, how it reaches the shader, and the jitter that appears when the origin is chosen badly. It is one stage of coordinate precision and projection on the GPU.

Where the significant digits go, before and after Two strips represent the 24 mantissa bits of a single-precision float holding a coordinate. In the absolute case a Web Mercator value near sixteen million spends 24 of its bits encoding the magnitude of the world position, leaving nothing for sub-metre detail — the low bits that would have carried centimetres do not exist. In the relative case, after a nearby origin has been subtracted, the value is a few hundred metres, so 9 bits carry the magnitude and 15 remain for detail, which resolves to a fraction of a millimetre. MANTISSA BIT 0 4 8 12 16 20 24 all 24 bits spent on magnitude Absolute x ≈ 16 700 000 nothing left below 2 metres magnitude detail — sub-millimetre Relative x ≈ 240 the same 24 bits, spent where the data is A 1024-metre origin grid leaves about 15 bits of detail — roughly 0.1 mm.
The float did not get better; the number got smaller. That is the whole of relative-to-eye encoding, and it is why the fix costs one subtraction rather than a wider type WGSL does not have.

Runnable reference implementation

Two pieces of code define the encoding: the host-side pack, and the uniform block plus vertex shader that consume it.

typescript
// One tile's worth of coordinates, encoded relative to that tile's origin.
// The subtraction happens in f64 — JavaScript numbers already are — and only
// the residual is narrowed to f32.
interface EncodedTile {
  originX: number;          // f64, kept on the host
  originY: number;
  local: Float32Array;      // interleaved residual x, y
}

function encodeTile(coords: Float64Array, tileMinX: number, tileMinY: number): EncodedTile {
  // Snap the origin to a coarse grid so nearby tiles share origins where
  // possible; this keeps the number of distinct matrices small.
  const originX = Math.floor(tileMinX / 1024) * 1024;
  const originY = Math.floor(tileMinY / 1024) * 1024;

  const local = new Float32Array(coords.length);
  for (let i = 0; i < coords.length; i += 2) {
    local[i]     = coords[i]     - originX;
    local[i + 1] = coords[i + 1] - originY;
  }
  return { originX, originY, local };
}

The shader side never sees a large number at all. The view-projection matrix is built on the host with the tile origin already folded in, so the vertex residual maps straight to clip space.

wgsl
struct Frame {
  // Built per tile on the host: view_proj * translate(origin - camera).
  local_to_clip : mat4x4<f32>,
};
@group(1) @binding(0) var<uniform> frame : Frame;

@vertex
fn vs(@location(0) local_xy : vec2<f32>) -> @builtin(position) vec4<f32> {
  // local_xy is at most a tile edge in magnitude — a few hundred metres.
  return frame.local_to_clip * vec4<f32>(local_xy, 0.0, 1.0);
}

The matrix carries the magnitude and the vertex carries the detail. That split is the entire technique, and the reason it works is that the matrix multiply happens in a coordinate system where the large translation has already been composed in f64 on the host, so the shader’s f32 arithmetic only ever operates on values that fit comfortably.

Parameter reference

Value Setting here Guidance
Origin grid 1024 m Coarser grids mean fewer distinct matrices and larger residuals. At 1024 m the worst residual is ~1448 m diagonally, where f32 resolves to 0.1 mm.
Residual type f32 f16 gives ~1 m at a 1024 m residual — too coarse. A normalised u16 over a known tile extent gives 2 cm and halves the buffer.
Origin storage f64 on the host Never uploaded as a pair of f32s; the whole point is that it stays wide until the matrix is built.
Matrix build per tile, per frame 64 bytes per tile per frame is negligible; building it per draw rather than per tile is the mistake.
Camera snap 256 m The camera-relative part of the origin should only change in steps, or every frame re-rounds the residuals differently and the geometry jitters.

The camera snap in the last row is the subtle one, and the next section is about why it exists.

Doing the subtraction on the right side of the narrowing Two lanes compare where the origin subtraction happens relative to the point at which coordinates are narrowed to single precision. In the wrong arrangement the coordinates are narrowed to f32 first, at fetch or decode time, and the subtraction then operates on values that have already lost their low bits, so the residual is correct to two metres and no better. In the right arrangement the coordinates stay double precision through the subtraction and only the small residual is narrowed, so every bit of the source precision survives. SUBTRACT, THEN NARROW Wrong order Right order Narrow to f32 at fetch or decode Subtract the origin bits already gone Residual is coarse no better than 2 m Keep f64 source stays wide Subtract the origin full precision Narrow the residual sub-millimetre If the server does the subtraction, the browser never sees a wide value at all — which is fine.
The two arrangements are one line apart in the code and an order of magnitude apart in the result. A fetch that returns a Float32Array has already made the choice for you, which is why the source type is the first thing to check.

The jitter that appears when the origin moves

A natural implementation makes the origin the camera position: subtract where the camera is, and residuals are small by definition. It works, and it produces a specific and confusing artefact — geometry that shimmers while the camera moves and is rock-steady when it stops.

The cause is that the residuals are recomputed against a different origin every frame. A vertex whose residual is 240.00003 metres this frame is 239.99998 next frame, because the origin moved by an amount that does not divide evenly into the f32 grid. Both values are correct to within the precision available; they simply round differently, and the difference is a fraction of a pixel of movement applied inconsistently across the geometry. The eye reads that as boiling.

There are two fixes and they compose. The first is to quantise the origin: snap the camera-derived origin to a grid coarse enough that it changes rarely — 256 metres is a reasonable choice — so the residuals are stable across the frames between snaps. The second is to make the origin per tile rather than per camera, which removes the dependency on the camera entirely: a tile’s origin is a property of the tile and never changes, so its residuals are computed once at load and are identical every frame thereafter.

Per-tile origins are the better default for a 2D map, because tiles already exist as a unit and the origin costs nothing to carry alongside the vertex range. Camera-relative origins earn their place in a globe view, where there are no tiles at the scale that matters and the visible extent is bounded by the horizon rather than by a grid.

Two origins, and which one wins

A pipeline that draws several layers over the same ground quickly acquires more than one origin, and the question of which one a given draw uses is worth settling explicitly rather than by whichever code path ran last.

The stable answer is that the origin belongs to the geometry, not to the pass. A vector tile encoded at load time against its own origin carries that origin for its whole life in VRAM; a point layer streamed from a different service carries whatever origin that service chose. Two layers drawn in the same frame can therefore have different origins, and that is fine — each one’s matrix is built from its own, and the two matrices differ only in a translation that was composed in f64.

What breaks is sharing a matrix. A frame that builds one local_to_clip and reuses it across layers is implicitly asserting that every layer shares an origin, and the symptom when they do not is an entire layer offset by the difference between the two — often a clean, suspiciously round number like 1024 metres, which is a useful diagnostic in itself.

The bookkeeping that avoids it is to store the origin in the same record as the buffer range, and to make the matrix a function of that record rather than of the frame. It costs one vec2 per layer and removes the whole class of bug.

Failure modes

  • Geometry boils while the camera moves. A camera-relative origin recomputed every frame. Fix: snap the origin to a coarse grid, or move to per-tile origins.
  • A visible seam between two tiles, growing with zoom. Two tiles used different origins and one was rendered with the other’s matrix. Fix: carry the origin in the same record as the vertex range, so they cannot be indexed independently.
  • Everything is correct until the data crosses the antimeridian. The origin was computed from a bounding box that wraps, so it lands halfway around the world from the data. Fix: compute origins per tile from the tile’s own key, never from a bounding box of the whole dataset.
  • The encoding made no difference. The subtraction was done after the coordinates had already been narrowed to f32 — often because the fetch produced a Float32Array. Detection: the input array’s type. Fix: keep the source in f64 until after the subtraction, or do the subtraction on the server.
  • Residuals are large and precision is still poor. The origin grid is too coarse relative to the data extent — a 100 km grid gives residuals where f32 resolves to about 1 cm, which is fine for a basemap and not for survey data. Fix: match the grid to the precision the data actually carries.
Choosing what the origin is relative to A comparison of three origin choices across three properties. A per-tile origin never changes, so residuals are stable and computed once at load, and it needs one matrix per tile. A camera position origin changes every frame, which makes residuals unstable and produces visible jitter, but needs only one matrix. A snapped camera origin, quantised to a 256-metre grid, changes rarely, keeps residuals stable between snaps, and also needs only one matrix. ORIGIN CHOICE · THREE PROPERTIES Changes Jitter Matrices Per tile never none one per tile Camera position every frame visible one Snapped camera rarely none one The one arrangement to avoid is the unsnapped camera origin — it is the only row with a visible artefact.
Per-tile origins are the right default for a tiled 2D map, because the tile is already the unit everything else is organised around. Snapped camera origins are the answer for a globe, where there is no tile at the relevant scale.

Backend / Python interop note

The subtraction is a vectorised column operation in numpy, which makes the server the cheapest place to do it. A tile endpoint that returns residuals plus an origin removes the loop from the browser entirely and shrinks the payload, because residuals compress far better than absolute coordinates — the high bits that used to differ between neighbouring points are simply gone.

python
import numpy as np

def encode_relative(xs: np.ndarray, ys: np.ndarray, grid: float = 1024.0):
    """xs, ys: float64 projected metres. Returns (origin, f32 residuals)."""
    origin_x = np.floor(xs.min() / grid) * grid
    origin_y = np.floor(ys.min() / grid) * grid
    local = np.empty((xs.size, 2), dtype=np.float32)
    local[:, 0] = (xs - origin_x).astype(np.float32)
    local[:, 1] = (ys - origin_y).astype(np.float32)
    return (float(origin_x), float(origin_y)), local

Two cautions when the origin comes from the server. It must be transported as a double — JSON numbers are f64, so a header or a metadata field is fine, but an f32 column in the Arrow schema is not. And it must be per tile: a single origin for a whole response defeats the purpose the moment the response spans more than a few kilometres. Putting the origin in the record batch metadata, one per batch, keeps the two properties together in a form the client reads once per batch rather than once per row.