Coordinate Precision and Projection on the GPU

WGSL has no double-precision float. Every coordinate a shader sees is an f32, which carries about seven significant decimal digits, and a Web Mercator metre coordinate near the edge of the world needs eight or nine before it is unambiguous. The result is not a rounding error in the abstract sense — it is buildings that shear, road centrelines that stair-step, and vertices that visibly snap to a grid whose spacing grows with distance from the projection origin. This page covers where the precision actually goes, the relative-to-eye encoding that recovers it, how Web Mercator is applied in a vertex shader once coordinates are small, and when a pipeline needs geocentric or local-tangent frames instead. It sits alongside the layout rules in memory alignment for spatial data buffers, and it decides where the tiles from texture and tile atlas management actually land.

Prerequisites

1. A device and a working render path. Nothing here needs an optional feature; every technique is arithmetic inside shaders that a baseline WebGPU device already supports.
2. Source coordinates in a known CRS. The techniques assume you know whether the incoming numbers are degrees, Web Mercator metres, or geocentric metres — mixing them is the most common cause of what looks like a precision bug.
3. A tile or chunk structure. Relative-to-eye encoding needs an origin to be relative to, and a tile is the natural unit.
4. Double precision available on the host. JavaScript numbers are f64, so every subtraction described here happens correctly before upload — which is exactly why the technique works.

Where the precision goes

An IEEE 754 single-precision float has 24 bits of mantissa, so it represents about 7.2 decimal digits regardless of magnitude. The absolute spacing between representable values therefore grows with the value itself, and that is the whole problem.

The last row is the one that matters, because 20 037 508 is exactly the extent of Web Mercator in metres. A city at longitude 150° sits near that value, so its vertices snap to a two-metre grid — while the identical data at longitude 0° snaps to a centimetre grid and looks perfect. That asymmetry is why the bug is so often reported as “the map is broken in Australia”.

Relative-to-eye encoding

The fix is not a wider type, because WGSL does not have one. It is to make the numbers small before they reach the shader, by subtracting a nearby origin on the CPU where f64 is available.

Concretely: pick an origin — the tile’s south-west corner, or the camera position — subtract it from every coordinate in double precision on the host, and upload the residual. A residual within a 512-metre tile never exceeds 512, where f32 spacing is 3 × 10⁻⁵ metres, which is thirty micrometres. The projection matrix then carries the origin, so the shader reconstructs world position by adding it back at a point where the arithmetic is happening in clip space and the magnitudes no longer matter.

// Host side: f64 arithmetic, then narrow. This subtraction is the whole trick.
function toTileLocal(
  coords: Float64Array,        // interleaved x, y in projected metres
  originX: number,
  originY: number,
): Float32Array {
  const local = new Float32Array(coords.length);
  for (let i = 0; i < coords.length; i += 2) {
    local[i]     = coords[i]     - originX;   // subtract in f64...
    local[i + 1] = coords[i + 1] - originY;   // ...then store as f32
  }
  return local;
}

struct Camera {
  // view_proj already has the tile origin folded in on the host, so the
  // shader never adds two large numbers together.
  view_proj  : mat4x4<f32>,
  // Residual of the tile origin relative to the camera, small by construction.
  origin_rel : vec2<f32>,
};
@group(0) @binding(0) var<uniform> camera : Camera;

@vertex
fn vs(@location(0) local_xy : vec2<f32>) -> @builtin(position) vec4<f32> {
  // Both terms are small; their sum is small; precision is preserved.
  let eye_space = local_xy + camera.origin_rel;
  return camera.view_proj * vec4<f32>(eye_space, 0.0, 1.0);
}

The rule that makes this work is simple to state: never add two large numbers in a shader. The origin is folded into the matrix on the host, the vertex is small, and the only addition in the shader is between two small quantities.

Projecting in the vertex shader

Once coordinates are small, the projection itself is cheap and can move onto the GPU. Web Mercator from longitude and latitude in degrees is two lines, and doing it per vertex means the buffers can hold geographic coordinates rather than projected ones — which matters when the same data feeds several projections.

const PI : f32 = 3.14159265359;
const R  : f32 = 6378137.0;          // WGS 84 semi-major axis, metres

fn web_mercator(lon_deg : f32, lat_deg : f32) -> vec2<f32> {
  let lon = radians(lon_deg);
  let lat = clamp(radians(lat_deg), -1.4844222, 1.4844222);  // ±85.051129°
  return vec2<f32>(R * lon, R * log(tan(PI * 0.25 + lat * 0.5)));
}

The clamp is not optional. Web Mercator is undefined at the poles — the logarithm diverges — and a latitude of exactly ±90° produces an infinity that propagates through the matrix multiply and makes the entire draw call disappear. Clamping to ±85.051129° is the standard convention and is what makes the projected world square.

Projecting on the GPU has a precision consequence worth naming: the output is a full-magnitude Mercator metre value, which is exactly the large number the previous section worked to avoid. So the two techniques compose in one direction only — project on the GPU when the result is immediately transformed by a matrix that brings it back to clip space, and pre-project on the host when the result has to be stored, compared or binned.

Geocentric and local-tangent frames

A globe view is a different problem. There is no plane to project onto, and the natural representation is geocentric Cartesian — ECEF, earth-centred earth-fixed — where a point is metres from the centre of the planet along three axes. Those values reach 6.4 million, which puts them squarely in the one-metre-spacing band of the table above.

The same fix applies with a different origin: subtract the camera’s ECEF position on the host and upload residuals. For a camera 500 km above the surface looking at a city, every visible vertex is within a few hundred kilometres of the camera, and the residuals sit comfortably in the range where f32 is accurate to millimetres.

A local tangent frame — east-north-up, anchored at a reference point — is the third useful representation, and it is what a site-scale visualisation should generally use. Converting ECEF to ENU is a rotation by the reference point’s latitude and longitude followed by the same translation, and once in ENU the coordinates are small by construction because they are metres from a nearby anchor.

The quantisation trap

There is a second way precision is lost that has nothing to do with f32, and it catches pipelines that have already done the origin subtraction correctly.

Vector tile formats quantise coordinates. The Mapbox Vector Tile specification, for instance, stores positions as integers in a fixed grid — 4096 units across a tile by convention — so a coordinate arriving from such a tile has already been snapped to roughly 4 metres at zoom 12 and 15 centimetres at zoom 17. No amount of care on the GPU recovers detail the tile never carried.

This matters because the two effects compound in a confusing way. A pipeline with an f32 problem shows artefacts that grow with distance from the projection origin; a pipeline with a quantisation problem shows artefacts that shrink as the user zooms in, because deeper tiles have finer grids over smaller areas. Seeing both at once — geometry that is coarse everywhere and additionally coarser in one hemisphere — is the signature of a pipeline that needs fixing in two places.

The practical response is to know the source grid and to stop optimising past it. If the tiles quantise to 15 centimetres, storing residuals as f32 at 30 micrometres is thirteen bits of precision that carry nothing, and a normalised u16 would carry the data exactly while halving the buffer. Matching the storage precision to the source precision is both cheaper and more honest about what the map actually knows.

Memory and performance implications

Precision work is almost free at run time and costs something at upload time. The subtraction is one pass over the coordinate array in f64 on the host, which is exactly the kind of per-record loop that Arrow-based streaming exists to avoid — so the right place for it is the server, where it is a vectorised column operation, or a worker, where it is off the main thread.

Storage improves. Residuals within a tile fit comfortably in f32 and often in f16 or a normalised u16: a 512-metre tile quantised to u16 has a resolution of 8 millimetres, which is finer than most source data, and halves the coordinate buffer. That is the same trade as any attribute compression, with the difference that here the precision was never available in the first place.

The one cost is per-tile uniform data. Each tile now carries its own origin, so a draw that batches many tiles needs the origin per instance rather than per pass — a vec2 per tile in an instance buffer, which is negligible, but it does mean the origins have to be uploaded and kept in step with the tiles they belong to.

Failure modes and diagnostics

• Geometry snaps to a grid, worse further from the projection origin. The classic f32 symptom. Detection: the artefact scales with longitude and vanishes near the prime meridian. Fix: subtract a per-tile origin on the host.
• A whole draw call disappears when the camera crosses the pole. A latitude of ±90° produced an infinity in the Mercator formula, which propagated through the matrix. Fix: clamp latitude to ±85.051129°.
• Adjacent tiles have a visible seam that grows with zoom. Two tiles used different origins and the shader added each to a matrix built for the other. Fix: make the origin travel with the tile, in the same instance record as its vertex range.
• Vertices jitter when the camera moves, but not when it is still. The origin is the camera position and is being updated every frame, so the residuals change every frame and the f32 rounding differs frame to frame. Fix: quantise the camera-relative origin to a coarse grid so it only changes when the camera has moved a long way.
• A globe view is accurate near the camera and wrong at the limb. ECEF residuals relative to the camera grow to thousands of kilometres at the horizon. Fix: per-tile origins rather than a single camera-relative one, exactly as in the 2D case.

Choosing where the projection happens

There are three places a projection can be applied, and the choice is usually made by accident rather than deliberately.

On the server, in the tile generator. The buffer holds projected metres, the client does nothing, and the data is bound to one projection forever. This is what most vector tile schemes do, and it is the right answer when the map only ever shows one projection — which for a Web Mercator basemap it does.

On the host, at load time. The buffer holds geographic degrees on the wire, the client projects once per tile into a typed array and uploads the result. This costs a pass over the coordinates in JavaScript, which is exactly the per-record work that a columnar pipeline exists to avoid, but it buys the ability to change projection without refetching. For an analysis tool that offers several projections it is often worth it — and it belongs in a worker, not on the main thread.

In the vertex shader. The buffer holds degrees, and every vertex is projected every frame. This costs a handful of transcendental operations per vertex per frame, which sounds expensive and generally is not: a log and a tan are a few cycles on hardware designed for exactly this, and the vertex stage is rarely the bottleneck in a map. What it buys is the ability to change projection between frames — for an animated projection transition, or a globe that flattens into a plane — with no data movement at all.

The precision interaction decides the rest. Projecting in the shader produces full-magnitude Mercator metres, so it composes with relative-to-eye encoding only when the projected value is consumed immediately by a matrix that returns it to clip space. If the projected value has to be stored, compared against a neighbour, or written to a buffer another pass reads, it needs to be small first — which means projecting earlier, with an origin, on the host or the server.

Continue in this section

• Relative-to-eye encoding for f32 coordinate precision — the subtraction, the uniform layout, and how the origin travels with the tile.
• Web Mercator projection in a WGSL vertex shader — the projection itself, the pole clamp, and when to project on the GPU at all.
• Converting ECEF and ENU frames in compute shaders — geocentric and local-tangent conversions for globe and site-scale work.

Testing precision without a globe

Precision bugs are hard to see on a screenshot and trivial to see in a test, and two cheap checks catch nearly all of them.

The first is a round-trip assertion. Take a handful of coordinates spread across the world — the antimeridian, a high latitude, the projection origin — run them through the host-side origin subtraction and the shader’s reconstruction arithmetic in double precision on the CPU, and assert the result is within a millimetre of the input. It is a unit test with no GPU in it, it runs in milliseconds, and it fails the moment someone changes which side of the boundary the origin subtraction happens on.

The second is a visual fixture at the worst case. Render a regular grid of known spacing at longitude 179.9°, at the deepest zoom the application supports, and compare against a stored image. Precision loss is a grid that stops being regular, which is one of the few artefacts that is genuinely obvious in a pixel diff — far more so than in the real data, where an irregular building footprint hides it.

Both belong in continuous integration rather than in a debugging session, because this is a class of bug that is introduced by refactoring — someone moves a subtraction, or changes a buffer from f64 to f32 one step too early — rather than by writing new code.

Related

• WebGPU architecture for spatial visualization — the section this topic belongs to.
• Memory alignment for spatial data buffers — how the coordinates are laid out once they are small.
• Texture and tile atlas management in WebGPU — the raster counterpart to this vector-side work.
• Python to GPU streaming with Arrow and GeoParquet — where the origin subtraction belongs.
• On-GPU viewport culling for vector tiles — the pass that consumes these coordinates first.