Debugging and Validation for Spatial GPU Pipelines

A CPU bug crashes or throws; a GPU spatial bug renders. An out-of-bounds storage read returns zero rather than faulting, an out-of-bounds write is discarded, a NaN coordinate produces a primitive that is silently culled, and a transform applied in the wrong frame yields a map that looks entirely plausible and is wrong. That failure profile is what makes debugging here a distinct discipline rather than an application of ordinary technique. This page covers the three tools that make a WebGPU pipeline speak — error scopes for validation failures, object labels so the messages name something recognisable, and the sentinel patterns that turn silent numerical corruption into a visible signal — and the order to reach for them. It sits under performance tuning and profiling.

Prerequisites

  1. A device with error handling installed before the first allocation. Scopes pushed after resources exist cannot catch their creation, so this composes directly with the device handshake.
  2. A build flag. Every technique here belongs in development and staging builds; two of the three are cheap enough to leave in production and one is not.
  3. A CPU reference for at least one kernel. Half of the value of the tooling below is having something to compare against.
  4. Familiarity with the ordering rules. Many apparent data bugs are ordering bugs, and the pipeline fundamentals page is where those rules live.
Triaging a GPU failure in one question A decision diagram. The question asks whether the failure produced any message at all. On the yes branch the failure is a validation, out-of-memory or internal error, which an error scope will capture and attribute, and the response is to read the message and fix the descriptor or shed resources. On the no branch the pipeline is producing valid API calls with wrong results, which no scope will ever catch, and the response is sentinel values and count assertions that turn the silence into a signal. DID IT SAY ANYTHING? Was there an error message? from a scope or the event Error scopes attribute it validation, OOM or internal yes Read it and fix the descriptor deterministic, reproducible Valid calls, wrong results no scope will help no Sentinels and count checks make the silence audible Most spatial pipeline bugs, once the descriptors are right, live on the right.
The two branches need completely different tools, and reaching for the wrong one is where the time goes. A scope that catches nothing is not a clean bill of health — it means the bug is on the right-hand branch.

Three channels, three meanings

WebGPU reports failure through three distinct channels, and treating them as one is the first mistake.

Validation errors are the application’s fault: a descriptor that does not satisfy the spec, a binding that does not match a layout, an offset that is not aligned. They are deterministic, reproducible and always fixable. device.pushErrorScope("validation") captures them.

Out-of-memory errors are a resource problem rather than a bug: an allocation the device cannot satisfy. They are captured by pushErrorScope("out-of-memory") and are the signal to shed resolution or evict, not to fix code.

Device loss is neither — the device is simply gone, and the response is the recovery path described under handling device lost.

Conflating the first two is the common error. Wrapping everything in a single scope and logging “GPU error” throws away the distinction between “this code is wrong” and “this machine is out of memory”, which are the two most different diagnoses available.

Channel How to capture What it means
GPUValidationError pushErrorScope("validation") The code is wrong; deterministic and fixable
GPUOutOfMemoryError pushErrorScope("out-of-memory") The device is full; shed or evict
GPUInternalError pushErrorScope("internal") The driver failed; usually a shader too complex to compile
device.lost the lost promise The device is gone; rebuild everything
uncapturederror event device.addEventListener Anything no scope caught — a safety net, not a strategy

Labels, and why they are not optional

Every WebGPU object accepts a label, and every error message quotes it. An unlabelled pipeline produces “Binding size is smaller than the minimum for [Buffer]”; a labelled one produces the same message naming tile-vertex-buffer-z14, which is the difference between a search and a fix.

The cost is a string per object, which is nothing, and the discipline is to label at creation rather than to add labels while debugging — because the moment they are needed is the moment adding them is most disruptive. A convention that includes the layer, the purpose and any distinguishing index ("cull-pipeline/vector", "readback/frame-2") makes messages self-describing.

Labels also flow into browser developer tools and into GPU captures, so they pay off twice: once in error messages and once when reading a frame capture where an unlabelled draw call is anonymous.

What each diagnostic costs and where it belongs A table of four diagnostics with their cost and the builds they belong in. Object labels cost a string per object, which is negligible, and belong in every build including production. Phase-level error scopes cost a small amount at start-up and belong in development and staging. Per-resource error scopes cost measurably more and belong in development only, when localising a specific failure. Sentinel buffer fills cost a clear per pass and belong in development and staging. DIAGNOSTIC · COST · WHERE Cost Belongs in Object labels negligible every build Phase error scopes small at start-up dev + staging Per-resource scopes measurable dev only, while hunting Sentinel fills a clear per pass dev + staging Gate the rest behind a build flag rather than deleting them once a bug is closed.
Only the first row is free enough to leave everywhere, and it is also the one with the highest ratio of value to effort — a labelled error is a fix and an unlabelled one is a search.

Sentinel values and the silent-corruption problem

The hardest bugs here produce no error at all, and the technique that finds them is to make the silence audible.

The pattern is to initialise every output buffer to a value the kernel cannot legitimately produce — a large negative number, or a specific bit pattern — before the pass runs. Any element still holding the sentinel afterwards was never written, which turns “some features are missing” into “elements 4096 to 4351 were not processed”, which is usually enough to identify a dispatch-size or tail-guard bug immediately.

The same idea applies to counts. Writing the element count the kernel believes it processed into a known slot, and comparing it against what the host dispatched, catches every case where a guard rejected work it should have accepted.

For coordinates specifically, NaN deserves its own treatment, because it propagates: one NaN in a transform matrix makes every vertex NaN, and a NaN vertex is silently discarded by the rasterizer rather than drawn wrongly. The result is geometry that vanishes with no message, and finding the source means testing for it explicitly — which is the subject of diagnosing NaN coordinates.

Where the tools cost something

Error scopes are not free. Each push-and-pop pair forces the implementation to track errors for the enclosed work, and wrapping every individual resource creation in its own scope measurably slows start-up on a large application. Wrapping phases — all pipeline creation, all buffer allocation, each frame’s submission — captures the same information at a fraction of the cost.

Labels are effectively free, and belong in production.

Sentinels cost a buffer clear per pass, which is real but small, and belong in development and staging only. The count-writing variant is cheap enough to leave enabled everywhere, and is the single highest-value diagnostic on this page for the amount it costs.

A sentinel fill turning silence into a location Two strips of sixteen output slots. The first shows the buffer after being filled with a sentinel value the kernel cannot legitimately produce. The second shows the same buffer after the dispatch: slots zero through eleven hold real results, and slots twelve through fifteen still hold the sentinel, which says precisely that the last four elements were never written — a tail-guard or dispatch-size bug, localised to an index rather than described as missing features. SLOT 0 2 4 6 8 10 12 14 16 sentinel value in every slot Pre-filled sentinel a value the kernel cannot legitimately produce real results still sentinel After the pass 12 of 16 slots 12–15 were never written — a tail guard or dispatch-size bug Cheap enough for staging; the count-comparison variant is cheap enough for production.
Without the sentinel the same run produces zeros in those four slots, which are indistinguishable from legitimate results. With it, the bug reports its own index range.

A CPU reference is the strongest tool here

Everything above makes a pipeline speak; a reference implementation makes it verifiable, and for spatial kernels it is worth the effort in a way it often is not elsewhere.

The reason is that spatial kernels have a property most GPU code does not: their inputs and outputs are small, structured and independently meaningful. A cull pass takes bounding boxes and a frustum and returns a survivor list. A binning pass takes centroids and a grid and returns counts. Both are twenty lines of readable JavaScript or Python, both run in milliseconds on a fixture of a few thousand elements, and both produce output that can be compared exactly rather than approximately.

The discipline that makes it pay is to write the reference first, or at least to write it from the specification rather than from the shader. A reference transcribed from the WGSL reproduces the WGSL’s bugs and asserts that the code does what the code does. A reference written from the intent asserts that the code does what it should.

Two fixture shapes catch most problems. One whose element count is an exact multiple of the workgroup size, and one that is deliberately not — because the entire class of tail-guard bugs only appears in the second, and a fixture of a round thousand elements at a workgroup size of 256 will never find them.

Where exact comparison is impossible — floating-point accumulation orders differ, and an atomic-based compaction has no defined order — compare the properties instead: the same set of survivors, the same total, the same count. That is weaker than exact equality and still strong enough to catch every structural bug.

Reading a frame capture

Browser developer tools and vendor capture tools both offer frame capture for WebGPU, and knowing what to look for turns them from an overwhelming wall of state into a quick answer.

Three things are worth checking first. The pass list: does the frame contain the passes it should, in the order it should, and is anything recorded twice? A duplicated pass is a surprisingly common bug in a framework integration where a lifecycle hook fires more often than expected. The bind group contents at each draw: does the buffer bound at each slot have the label expected, and the size expected? And the draw parameters: is the instance count what the application believes it is, particularly where it comes from an indirect buffer.

Those three answer most “why is nothing rendering” questions before any shader is examined, which is the right order — a shader that is never dispatched, or dispatched with a count of zero, produces exactly the same blank screen as one that is wrong.

Making a bug reproducible before fixing it

A GPU bug that only appears sometimes is usually a bug that depends on state the application is not controlling, and the fastest route to a fix is often to remove that dependence rather than to reason about it.

Three sources of variability account for most of it. Timing: a race between an upload and a pass shows up on a slow network and not a fast one, and throttling the connection in developer tools makes it deterministic. Data: a dataset that happens to contain a degenerate polygon, a duplicated vertex or a NaN triggers a path a clean fixture never reaches, so capturing the actual failing payload matters more than reproducing the interaction. And ordering: an atomic-based compaction produces a different array on every run, so a bug in whatever consumes it appears intermittently.

The response to each is the same shape — pin the variable. Throttle the network, save the payload, switch to the deterministic compaction path. A bug that reproduces every run is usually most of the way to being understood, and the pinning takes minutes.

The one to resist is adding a delay until the problem goes away. A setTimeout that makes a race disappear has not fixed anything; it has widened a window, and the window will narrow again on faster hardware or a slower network. When a delay changes the behaviour, that is a diagnosis — something needs a fence or a barrier — rather than a fix.

Failure modes and diagnostics

  • An error appears with no context. No labels. The fix is a convention, applied at creation.
  • uncapturederror fires but no scope caught it. The scope was popped before the work was submitted; validation for a submission is reported at submit time, not at record time.
  • A scope catches nothing and the bug persists. The failure is not a validation error — it is a logic bug producing valid API calls. Sentinels rather than scopes.
  • Geometry vanishes with no error. A NaN in the transform or the vertex data, silently discarded by the rasterizer.
  • The bug disappears when the profiler is enabled. A synchronisation bug hidden by the extra ordering the profiler introduces. That is information: it points at a missing barrier or a missing fence.

Building diagnosis into the pipeline

The techniques above are reactive; a few structural choices make a pipeline diagnosable before anything goes wrong, and they cost little enough to adopt as defaults.

One buffer, one purpose, one label. A buffer reused for two things across passes saves an allocation and destroys attributability: an error naming it says nothing about which use was wrong, and a sentinel check cannot tell which pass failed to write. Separate buffers with descriptive labels are worth the memory.

Counts alongside data. Every output buffer that has a variable length should carry that length in a known slot, written by the pass that produced it. That single number is what lets a later pass, a readback or an assertion detect a mismatch, and it is the cheapest diagnostic available.

One place where each kind of resource is created. A createBuffer scattered across twenty files makes an allocation audit impossible; a single allocator function makes it a matter of adding a counter. That is the same registry idea the framework-integration pages use for lifecycle, applied to observability.

A deterministic mode. Where a pipeline uses atomics for compaction, having a build flag that switches to the scan-based deterministic version makes a whole class of bug reproducible. It is slower and it is only for debugging, and the day a compacted array has to be compared against a fixture it is the difference between a five-minute answer and a day.

When the bug only appears on one machine

Spatial GPU code has an unusually high rate of failures that reproduce on one vendor and not another, and it is worth knowing which categories those fall into before assuming a driver bug.

Barrier bugs are the largest category. Hardware that executes a workgroup in lockstep hides a missing workgroupBarrier, so code written and tested on one architecture fails on another with a wider or narrower wavefront. The tell is that the failure is in a kernel using workgroup memory and the results are wrong rather than absent.

Precision differences are the second. WGSL permits some latitude in transcendental function accuracy, so sin, log and pow can differ in the last bits between implementations. That never matters for shading and can matter for a comparison — a point exactly on a tile boundary landing in different cells on different machines.

Limit differences are the third and the most tractable: a buffer size, a workgroup count or a texture dimension that fits on the development machine and not on the target. Those are visible in the capability record, which is the argument for logging it with every bug report.

Genuine driver bugs exist and are rarer than they are blamed. Before concluding one, it is worth running the same kernel with a smaller workgroup size and with the deterministic compaction path, because both of those changes make the first two categories disappear.

Continue in this section

A checklist for a blank map

Nothing renders is the most common report and the least informative, so it is worth having an ordered checklist that narrows it in a couple of minutes rather than an afternoon.

First, is anything being submitted at all? A frame with no submit call renders nothing and raises nothing. Logging a submission counter for one second answers it.

Second, is the draw count non-zero? An indirect draw whose instance count was never written is a legal draw of zero instances. Read the indirect buffer back once and look at it.

Third, are the bind groups pointing at the buffers you think? A capture answers this immediately, and the labels are what make the answer readable.

Fourth, is the geometry where the camera is? A transform in the wrong frame puts a perfectly valid scene several thousand kilometres away. Rendering a single known point at the camera’s own position is a five-line test that separates “nothing is drawn” from “everything is drawn somewhere else”.

Fifth, is anything NaN? One NaN in a matrix removes the entire scene silently, which is why it belongs on a checklist rather than at the end of a debugging session.

Only after those five is it worth reading a shader. Four of the five are answered by tooling this page has already described, which is the argument for putting that tooling in before it is needed rather than after.