May 2026 at AppX: 18 wins from the persistence-rearchitecture month
The headline: we found a structural bug in agent-turn persistence (the 0ms-zombie), fixed it across five engineering waves, AND launched this engineering blog, AND shipped a dozen smaller polish items.
AppX team ·
May was a persistence month. Early on we found a structural bug in how AppX recorded agent turns — a kind of row that could get stuck mid-flight if the backend was killed at exactly the wrong moment. We named it the 0ms-zombie, and chasing it turned into a full rearchitecture of the chat_turns table across five shippable waves. While that was happening, we also tuned Forge under load, refreshed the canvas chat UI to Bloom-parity, closed a handful of operational paper cuts, and launched this engineering blog. Eighteen wins, grouped by theme.
1. Persistence rearchitecture
The 0ms-zombie, killed. Agent-turn rows could survive a backend SIGKILL in an in-flight state forever — duration 0ms, status mid-turn, nobody owning them. The fix was structural, not a patch: a single-row-per-turn invariant, a database CHECK constraint to make the bad state unrepresentable, and a sweeper that runs on backend startup as the moral equivalent of a SIGKILL handler. Deep dive: the 0ms zombie.
Typed columns over metadata JSON. Nineteen fields moved out of a metadata JSONB blob and into proper typed columns, with a composite index for the hottest read pattern. The chat history list query no longer has a single JSON_EXTRACT on its hot path. Deep dive: typed columns over metadata JSON.
Raw SQL banned in templates. Inside our tagged `sql`` template literals, column references now have to come from the typed query builder. No more bare strings drifting away from the schema. An entire class of schema-drift bugs becomes a compile-time error. Deep dive: ban raw column references.
Wave-shaped refactor. The whole migration shipped as five independently revertible waves, with shadow-mirror writers comparing old and new storage in production before the kill-switch wave flipped reads over. No big-bang migration, no freeze, no surprise on the day of cutover. Deep dive: wave-shaped refactors.
2. Forge stability
Warm pool tuned. Forge — our sandbox layer that hands every project a real container with Expo running — now holds around eighty warm containers per 24 GB VPS. The trick was aggressive sharing of node_modules across containers via mount, so the marginal cost of an idle slot is tiny. Cold-to-QR stays under two seconds at this density.
Webhook-driven state. Forge used to tell the backend what was going on via polling, which meant a slow drift between Docker reality and our database. We replaced the polling loop with signed webhooks on every state transition. Propagation time went from five-to-ten seconds to roughly one millisecond, and an entire class of "the dashboard disagrees with itself" bugs disappeared with it.
3. Chat polish
Editorial Dusk redesign. Eight canvas components got restyled in one pass — chat bubble, phone preview panel, engine activity card, ask-user card, input area, header, turn summary, bot avatar. New token system, new type stack (Bricolage Grotesque, Instrument Serif, JetBrains Mono on a midnight palette), and a signature trailing-underline element that ties the surface together. Net delta: minus six hundred lines of CSS-via-className.
Paused-turn race, fixed. A 112-millisecond window between when the orchestrator saved a checkpoint and when it marked a turn "active" was wide enough for a chat:question-expired event to fire against a turn that wasn't yet known to be paused. The fix is one line in the right place: set active-turn state immediately after the checkpoint write, on both pause sites. Caught by a real user who hit refresh at exactly the wrong moment.
Echo guard for trivial input. Typing "hi" used to occasionally produce a clarifying question — the architect dutifully asking the user to clarify "hi." Fixed at two levels: a hard rule in the prompt, and a bigram-overlap guard that rejects an architect question whose content is too close to the user's input. Belt and suspenders, on purpose.
AskUserCard polish. When the architect chose the non-destructive branch (asking instead of editing), the question used to render as bare prose in the message stream. Now it renders inside the proper card frame, with the same affordances as the destructive branch. Small thing, large clarity win.
4. Operations and reliability
MinIO buckets-init retry. Backend startup used to occasionally fail if MinIO wasn't quite ready yet. Now it retries with exponential backoff up to five attempts and escalates with a useful error message instead of a stack trace.
Gitea startup probe split. The startup health check for our internal git service used to log "Gitea is down" for anything non-200. It now distinguishes 401/403 (Gitea is up, auth is misconfigured) from genuine connection failure. Less log noise, faster diagnosis.
Signed webhooks end-to-end. Forge webhooks are now HMAC-signed with a shared secret, and the backend rejects anything unsigned or with a mismatched signature. Closing an obvious gap before someone else found it.
Smoothed out a deploy-pipeline drift class. A subtle pattern where two different deploy paths could leave our orchestration layer and our recorded config out of sync. Resolved by routing all deploys through the canonical API path. Bonus: deploys are now reproducible from the recorded config alone.
5. Engineering blog launch
AppX Engineering went live. Eighteen posts across five rounds, covering everything from prompt-injection defenses and agent-loop hardening to the Forge architecture deep-dive and the persistence rearchitecture above. The blog is partly for ourselves — writing things down forces clarity — and partly an open invitation for other small teams building agent systems to crib whatever's useful.
That's May. Five waves of persistence work, a measurably more stable Forge, a chat surface that finally looks like the product we want it to be, and a blog to talk about the engineering underneath it all. If you're curious about any of the deep-dives, they're at /blog.