Handling Device Lost and Recreating GIS Resources

A GPUDevice can be lost at any moment — a driver reset, a background tab discarded under memory pressure, an external GPU unplugged, or the application’s own call to destroy(). When it happens, every buffer, texture, pipeline and bind group created from that device becomes invalid at once, and a map that treats the loss as a crash simply disappears. Recovery is mechanical if the application was built for it and close to impossible if it was not, because it requires knowing what to rebuild and in what order. This page is that recovery path, the one loss reason that must not be retried, and the resource registry that makes rebuilding a single function. It is one stage of initializing WebGPU devices for GIS workloads.

The branch that stops a recovery loop A decision diagram on the reason a device was lost. When the reason is destroyed, the application asked for the teardown, so the correct response is to leave it torn down and re-acquire nothing. When the reason is unknown — a driver reset, a discarded tab, a GPU removed — the correct response is to re-acquire a device once, rebuild the capability record from the new adapter, and then run every registered rebuilder in order. DEVICE LOST · READ THE REASON device.lost reason resolves once, never rejects destroyed — we asked leave it torn down yes Re-acquire nothing retrying fights the app unknown — involuntary driver reset or discard no Re-acquire, then rebuild capability record first A guard flag keeps several subsystems from each starting their own recovery.
Both branches are one line of code and the difference between them is a map that recovers and a map that spawns devices until the tab dies.

Runnable reference implementation

device.lost is a promise that resolves — never rejects — exactly once. The reason distinguishes a deliberate teardown from an involuntary one, and the two need opposite responses.

typescript
type Rebuild = (device: GPUDevice) => void | Promise<void>;

class DeviceHost {
  private device: GPUDevice | null = null;
  private readonly rebuilders: Rebuild[] = [];
  private reacquiring = false;

  /** Every subsystem registers how to rebuild itself, once, at start-up. */
  onRebuild(fn: Rebuild): void {
    this.rebuilders.push(fn);
  }

  async acquire(): Promise<GPUDevice> {
    const adapter = await navigator.gpu?.requestAdapter({
      powerPreference: "high-performance",
    });
    if (!adapter) throw new Error("no adapter — route to the fallback renderer");

    const device = await adapter.requestDevice({
      label: "spatial-device",
      requiredLimits: { maxBufferSize: 256 * 1024 * 1024 },
    });
    this.device = device;

    void device.lost.then((info) => {
      // "destroyed" means we asked for this. Re-acquiring would resurrect a
      // map the application deliberately tore down.
      if (info.reason === "destroyed") return;
      void this.reacquire();
    });

    return device;
  }

  private async reacquire(): Promise<void> {
    if (this.reacquiring) return;          // one recovery at a time
    this.reacquiring = true;
    try {
      const device = await this.acquire();
      for (const rebuild of this.rebuilders) await rebuild(device);
    } finally {
      this.reacquiring = false;
    }
  }
}

Every subsystem — the tile cache, the layer renderers, the profiler — registers one rebuild function and nothing else. That is what keeps recovery to a single code path: the host does not know what a tile cache is, and the tile cache does not know that a device was lost, only that it has been handed a new one and must rebuild against it.

Parameter reference

Value Setting here Guidance
Re-acquire on "destroyed" never The application asked for the teardown. Retrying fights it, and the symptom is a device count that only grows.
Re-acquire on "unknown" yes, once Driver reset or tab discard. A single attempt, then the fallback path if it fails.
Concurrent recoveries one A guard flag; several subsystems noticing the loss must not each start a recovery.
Rebuild order registration order Register the device-level resources first so later rebuilders can depend on them.
Backoff before re-acquiring 500 ms A driver that has just reset may not be ready; the polling loop already implements the schedule.
What a device loss takes with it A table of six kinds of state and whether each survives a device loss. Buffers and textures do not survive. Pipelines and bind groups do not survive. The capability record does not survive in the sense that it must be rebuilt, because the new adapter may report different limits. The tile cache index survives. Decoded tile bytes survive if the application kept them. Camera and layer configuration survive. SURVIVES A DEVICE LOSS? Survives Buffers and textures no Pipelines and bind groups no Capability record rebuild it Tile cache index yes Decoded tile bytes if you kept them Camera and layers yes Host memory is cheap relative to VRAM — this is one of the few places to spend it deliberately.
The fifth row is the one worth designing around. Keeping decoded bytes on the host costs ordinary memory and turns recovery from a reload into a flicker, because nothing has to be fetched again.

What survives a loss, and what does not

The distinction that makes recovery tractable is between GPU-side state, which is gone, and CPU-side state, which is not.

Gone: every GPUBuffer, GPUTexture, GPUSampler, GPUBindGroup, GPUShaderModule, GPURenderPipeline and GPUComputePipeline. Also the capability record, because the new device may have different limits — a laptop that switched from discrete to integrated graphics on battery hands back a materially weaker device, and every buffer size derived from the old record is now wrong.

Not gone: the tile cache’s index, the decoded tile bytes if they were kept, the camera, the layer configuration, the fetch queue, and every piece of application state. This is the reason a good recovery is fast: the expensive part of loading a map is the network and the decode, and neither has to happen again.

The practical consequence is a rule about where decoded data lives. A pipeline that uploads a tile and immediately drops the source bytes has to refetch on recovery; one that keeps the bytes in an in-memory cache can rebuild the whole scene without a single request. Keeping them costs host memory, which is far cheaper than VRAM, and turns a device loss from a visible reload into a barely perceptible flicker.

Testing recovery before it happens in production

Device loss is rare enough in development that a recovery path written once and never exercised is a recovery path that does not work. Two ways of provoking it deliberately are worth wiring into a development build.

The first is device.destroy(). It resolves device.lost with reason "destroyed", which exercises the branch that must not re-acquire — the one whose failure mode is an infinite loop. Binding it to a key in a debug build takes a minute and catches the loop immediately.

The second is harder and more valuable: exercising the "unknown" branch. There is no standard way to force it, but the practical substitute is a fault-injection flag in the DeviceHost that synthesises the same recovery — tear down the current device, acquire a new one, run every rebuilder — without waiting for the driver to cooperate. It is not a perfect simulation, because a real loss also invalidates objects the application may still be holding, but it exercises the rebuild order and catches the two most common bugs: a rebuilder that captured the old device in a closure, and a layer that never registered one at all.

Both belong behind a flag rather than in tests, because the assertion that matters is visual. The map should come back with the same tiles, the same camera and no visible gap, and that is something a person confirms in a second and a test suite struggles to express.

Failure modes

  • The map disappears and never returns. Nothing listened to device.lost. Detection: the console shows validation errors referencing a destroyed device, and they stop when the tab is reloaded.
  • Two devices exist after a recovery. Several subsystems each started a recovery. Fix: the guard flag, and one owner for the device.
  • Recovery loops forever. The application called destroy() and the handler re-acquired, which produced a device that was then destroyed again. Fix: branch on reason === "destroyed".
  • The map returns but is missing a layer. That layer registered no rebuilder, or registered one that captured the old device in a closure. Fix: rebuilders take the new device as a parameter and must never reference the outer one.
  • Buffer sizes fail on the rebuilt device. The capability record was reused rather than rebuilt, and the new adapter is weaker. Fix: rebuild the record first and derive every size from it.
The order a recovery rebuilds in Four ordered stages. First a new device is acquired and the capability record is rebuilt from the new adapter, because every size downstream is derived from it. Second the pipelines and shader modules are recreated, since they are device-scoped and everything else binds against their layouts. Third the buffers and textures are recreated at sizes derived from the fresh record, and refilled from host-side bytes where those were kept. Fourth the bind groups are rebuilt last, because they reference both the layouts and the resources created in the two previous stages. RECOVERY ORDER · FOUR STAGES 1 Device and capability record limits may have changed every size derives from this 2 Pipelines and shader modules device-scoped, rebuilt fresh layouts come from here 3 Buffers and textures refill from host-side bytes no refetch if they were kept 4 Bind groups last reference layouts + resources they depend on both above Register rebuilders in this order and the recovery is one loop over a list.
The order is forced by the dependency graph rather than chosen. A rebuilder that creates a bind group before its pipeline exists fails, and the failure is attributed to the bind group rather than to the ordering.

What the user should see

Recovery is also a user-interface decision, and the default — say nothing and rebuild — is usually right.

A device loss that recovers in a few hundred milliseconds is best handled silently. The map goes blank for a frame or two and comes back with the same view; showing an error toast for that is worse than showing nothing, because it draws attention to a fault the user would otherwise not have registered. Log it, count it, and move on.

A recovery that takes longer — because tiles have to be refetched, or because the driver is slow to hand back a device — deserves the same treatment a slow initial load gets: keep the last frame on screen rather than clearing to blank, and let the tiles fill in as they arrive. That is a matter of not destroying the canvas contents, which is the default behaviour, and of not clearing application state that the recovery does not need to discard.

A recovery that fails, twice, is the case that needs telling. At that point the honest response is the fallback renderer described under browser support and fallback routing, with a one-line note that hardware acceleration is unavailable — because the map the user then sees is genuinely a different, more limited thing, and pretending otherwise makes its limits look like bugs.

Backend / Python interop note

Device loss is a client-side event, but it has one server-side consequence worth planning for: a recovery that has to refetch will do so as a burst, and a burst from many clients at once — a driver update rolling out across a fleet, for instance — looks exactly like a traffic spike.

Two things make that harmless. The first is idempotent, cacheable tile URLs, so a refetch after recovery is served from the browser’s HTTP cache rather than from the origin; the same property that makes VRAM eviction cheap makes device recovery cheap. The second is to keep decoded bytes on the host, which removes the refetch entirely for anything already loaded.

For a Python tile service the practical guidance is unchanged from ordinary caching hygiene — immutable URLs with a version in the path, long max-age, and an ETag — but it is worth noting that this is the case where it matters most, because it converts a fleet-wide recovery event from a thundering herd into a no-op.