Forge deep-dive: webhook-driven state, zero polling, no drift
Sequel to why-we-built-forge. Polling produces a structural drift window. HMAC-signed webhooks on every sandbox transition take state propagation from 5-10s to ~1ms. Idempotent handler, state-machine guard, replay-on-restart, and one slow reconciler kept as safety belt.
AppX team ·
A previous post on this blog — why we built Forge — covered the build-vs-buy decision behind AppX's per-user app sandbox layer. Start there if you want the context for what Forge is and why it exists. This post is a sequel. It goes one layer deeper into a specific design decision: how the backend knows what state any given sandbox is in.
The short version of the answer is "the agent tells it, the instant the state changes, over HMAC-signed HTTP." The long version is more interesting, because we did not start there. We started with polling, and the failure modes of the polling design are the reason this post exists.
The polling design we had first
The first version was the obvious one. Forge's node agent owns the Docker daemon on each host. The backend wants to know the state of every sandbox. So the backend asks: every 5 to 10 seconds, the backend hits the orchestrator's API and says "what state is sandbox X in?" The agent looks at Docker, returns the current state, and the backend writes it to the sandbox database. Done.
This is the design 90% of teams ship first. It's how most "is this thing healthy yet" checks work in production. It is also subtly, expensively wrong for a system where users are watching the result land on their phone.
The drift class polling produces
sequenceDiagram
participant S as Sandbox
participant B as Backend
S->>S: state changes (running → stopped)
Note over B,S: drift window — backend still believes "running"
B->>S: poll (every N seconds)
S-->>B: learns the truth, late
Here is the failure mode. A container transitions from provisioning to running at t = 12.3 seconds after the request lands. The next poll happens at t = 15 seconds. For 2.7 seconds, the database believes the sandbox is provisioning and the truth is that it is running. That window is small. It is also enough to produce three different bad outcomes simultaneously.
t=10.0s user clicks "Generate"
t=10.1s backend claims a warm slot DB: provisioning
t=12.3s Docker reports container up DB: provisioning <-- DRIFT WINDOW
t=13.0s user's phone hits preview URL DB: provisioning <-- proxy 502s
t=14.0s concurrent request lands DB: provisioning <-- double-claim risk
t=15.0s poll fires, DB updates DB: running <-- finally correct
The reverse proxy routes based on the database column. During the drift window, it returns 502 to the user's phone because the row still says provisioning. The frontend "pull to refresh" UX shows the user a stale "still preparing" message because the API reads the same column. A concurrent backend request — say, a parallel chat edit landing in the same instant — sees the slot as unready and may try to claim a second one.
Three different bugs, one root cause: the database is a lagging mirror of reality, and the lag is sized by the poll interval. You can shorten the interval. You cannot eliminate the window. And shortening it pushes hundreds of state-query calls per second through the orchestrator's API for sandboxes that are not transitioning. Polling solves nothing and costs continuously.
Webhook-driven state
sequenceDiagram
participant S as Sandbox
participant B as Backend
S->>S: state changes
S->>B: signed webhook — immediately
B->>B: verify HMAC, apply transition
Note over B,S: no drift window
The replacement is exactly as boring as it sounds. The agent fires an HTTP POST to the backend the instant a Docker transition lands. Backend receives it on the order of one millisecond later. The DB row updates. No window.
{
"sandbox_id": "snd_a1b2c3",
"from": "provisioning",
"to": "running",
"at": "2026-05-16T12:00:00Z",
"node_id": "node-04"
}
Every transition the system cares about — provisioning → running, running → sleeping, sleeping → destroyed, plus the error paths — produces one of these. The agent watches Docker events directly, so the emit-to-truth gap is the latency of reading from the Docker event stream, not a polling cadence chosen by humans.
This is the kind of change that reads as a footnote in a design doc and is the single largest source-of-truth improvement we have ever shipped.
HMAC signing
A webhook endpoint is, by default, an unauthenticated POST handler that any process on the internet can reach. That is unacceptable for an endpoint whose job is to declare facts about state that downstream code trusts unconditionally.
The fix is standard and worth naming out loud. The agent and the backend share a secret. Every webhook body is signed with HMAC-SHA-256 over the raw bytes; the signature ships in a header. The backend recomputes the signature on receive and rejects anything that doesn't match. Unsigned or wrongly-signed POSTs return 401 without touching state.
The shared secret rotates on a calendar cadence. The rotation is bilateral — both sides cut over together — which keeps the signing surface atomic and avoids the brief "neither side knows which secret to use" hole that one-sided rotation produces.
Idempotency
Webhooks retry. They should — that is the resilience property that makes them better than polling in the first place. But retries mean the same transition arrives at the backend more than once, and "transition arrived" cannot be the trigger for state-changing work. The handler has to be idempotent at the row level.
The whole trick is one SQL statement.
UPDATE sandboxes
SET state = $new_state, updated_at = NOW()
WHERE id = $sandbox_id
AND state != $new_state;
If the row is already in the target state, the WHERE clause matches zero rows, the UPDATE is a no-op, and the affected-row count tells the handler this transition was already applied. Side effects keyed off the transition — emitting events, waking listeners, updating the reverse proxy — fire only when the affected count is 1. Replays are free.
State-machine guard (isValidTransition)
Not every transition is legal. A sandbox cannot go from destroyed to running — that would require the agent to be lying or confused. The backend keeps a small allow-list of legal (from, to) pairs and rejects anything not on it.
provisioning -> running OK
running -> sleeping OK
sleeping -> running OK
sleeping -> destroyed OK
destroyed -> running REJECTED
running -> provisioning REJECTED
Invalid transitions return a 4xx, log a WARN, and increment a metric. The metric is the actual point of the guard. Bugs in the agent show up as a spike on the chart, not as silent corruption in the state column. Catching a state-machine bug at the boundary is dramatically cheaper than catching it three days later when a reconciler finds an impossible row.
Recovery on backend restart
Webhooks have one failure mode the polling design did not: if the backend is down when a transition fires, the event is lost. Retries from the agent close most of that, but a long enough restart can drop transitions on the floor entirely. Worse, transitions that fire during the down window vanish without a trace.
The fix is a replay endpoint. When the backend comes back up and the agent reconnects, the agent dumps the current state of every active sandbox it owns in one call. The backend reconciles each row: if its DB disagrees with the agent's truth, the agent wins. Without this, a single restart leaves orphan rows in the DB forever, drifting further from reality with each subsequent transition.
The reconciliation runs through the same idempotent UPDATE used for individual webhooks. The state-machine guard still applies. The only thing that changes is the trigger.
What polling still does (defense-in-depth)
We did not actually delete polling. We demoted it. There is still a background reconciler that compares the DB's view of the world against the agent's view every few minutes, finds disagreements, and corrects them.
This is not the primary mechanism. It is a safety belt. If a webhook is dropped during a network partition, if the backend crashes mid-handler, if HMAC validation fails because a clock skew pushed the signed timestamp out of tolerance — the reconciler eventually catches the drift and repairs the row. Webhooks deliver correctness in normal operation. The reconciler delivers convergence under failure.
This is the same defense-in-depth shape we've used in other parts of the system: a fast primary mechanism whose failure modes are caught by a slower, exhaustive secondary mechanism. Each layer's job is to make the next layer's failures rare.
The numbers
Webhook round-trip from "Docker transition lands on the host" to "DB row reflects truth" is on the order of one millisecond typical, ten milliseconds at the p99. The polling interval it replaced was 5 to 10 seconds. That is a 5,000x to 10,000x latency improvement on state propagation. The cost is roughly fifty lines of agent code, a signed POST handler on the backend, and the idempotent UPDATE above.
The cost of getting it wrong, by contrast, was a continuous trickle of 502s on freshly-warm preview URLs, a stale-state UX bug nobody could quite reproduce in dev, and a class of double-provision incidents that only showed up under load. None of those bugs filed themselves under "polling design." They filed themselves under "preview is flaky." Misattributing infrastructure failures to the product is one of the most expensive mistakes a small team can make.
Closing reflection
The interesting thing about the webhook design is not that it is novel. It is not. It is the obvious correct design for any system where state changes are events the producer can name. The interesting thing is that we built the polling version first anyway, because polling is the default mental model for "how does service A find out about state in service B" and you have to actively reach for the better answer.
Forge's state model is now event-shaped end to end. The agent emits events. The backend's database is the materialized view of those events. The reverse proxy reads the same view. The frontend reads the same view. No component reaches across the boundary to ask the agent "are we there yet." The agent tells everyone, once, the instant the truth changes, signed.
Plumbing should not be exciting. Most weeks, this one isn't.