The screenshot that triggered a 5-wave rearchitecture
'Done · 0ms' looked cosmetic. Eight hours of investigation revealed it was a structural bug class. A week of work shipped a 5-wave persistence rearchitecture. The narrative of how a UI screenshot turned into the biggest backend project of the month.
AppX team ·
A user sent us a screenshot at 9am. Two cards in the chat panel of their AppX project. The first said Done · 0ms. The second said Done · Worked for 0ms. Both were attached to assistant messages that, when you clicked into them, were empty. The user's note was three words: "this looks broken."
I opened the screenshot, sighed, and put an hour on the clock. We'd fix it before lunch. Eight hours later I'd identified a structural class of bug. A week later we'd shipped a five-wave rearchitecture of our entire agent-turn persistence layer. This is the story of how a cosmetic-looking screenshot turned into the biggest backend project of the month.
First hypothesis: cosmetic
Both cards appeared after a page refresh, per the user's follow-up. That mattered: the turn ran, the user saw it complete normally, then they reloaded and the rehydrated version rendered as Done · 0ms. The live-stream path was fine. The hydrate path was buggy.
I found the hydrator in under a minute. Reasonable code. When a row came back with no terminal outcome set, the hydrator defaulted to a renderable shape: outcome = 'complete', completedAt = startedAt. The duration calc downstream did completedAt - startedAt, got zero, and the UI rendered Done · 0ms.
I drafted the clamp in my head: if durationMs <= 0, render null instead. Ten lines. I started typing. Then I paused.
The pause: why is outcome NULL?
The hydrator had a fallback for outcome IS NULL. Why was it ever NULL?
Thirty seconds of SQL later I had my answer, and I did not like it. Rows in the turns table with no terminal outcome, no body, creation timestamps from minutes ago. Not test data. Not migration debris. Real production rows from real recent turns. The hydrator wasn't lying. It was rendering exactly what was in the database. The database was the one lying.
A clamp would have hidden the 0ms symptom. It would not have addressed the fact that we had real users with real turns the database thought were still in progress, forever. I closed the frontend file and opened the backend.
Tracing the writer
The writer was a two-phase pattern. The kind every agent-loop persistence layer reaches for the first time someone asks "how do we render streaming turns?"
Phase one, at turn start: INSERT a placeholder row. Outcome NULL, body empty, started-at set. The frontend uses this to render a "thinking" bubble.
Phase two, at turn end: UPDATE the row to its terminal state. Set the outcome, fill in the body, write the completed-at timestamp.
Between phase one and phase two is a window. Inside that window, the agent loop is running: calling tools, streaming tokens, doing the actual work. If the process dies in that window — a deploy, an OOM kill, a node falling over, a network partition severing the update — the placeholder stays alive forever. Outcome stays NULL. The hydrator finds it later, defaults the missing fields, and renders Done · 0ms.
It was, in retrospect, completely obvious. Each layer in isolation was correct. The bug only existed in the join between them — in the silent assumption, baked into the schema, that the placeholder would always be merged before anyone read it. A deploy mid-turn produced this shape. We did multiple deploys a day.
The realization: it's a class, not a bug
The thing that turned this from a bug ticket into a project was the second-order question: how many other tables do we have that look like this?
Job queues. Chat messages. Tool-call records. Container provisioning state. Every place we did "insert pending, update to terminal" had the same window. Every one of them could produce orphans. Some of them probably already had.
By around noon I understood it as a class: TOCTOU on row state, where the time-of-insert and the time-of-terminal-merge are separated by an arbitrarily long agent loop, and the gap is adversarial. Deploys land in it. OOMs land in it. Crashes land in it. The window is small in absolute terms and unbounded cumulatively — at scale, with enough deploys, the row count of zombies trends monotonically up.
The fix could not be one ternary in one hydrator. It had to address the shape of two-phase persistence, in this table and every table like it.
Two fixes, one week
Two fixes on different timelines, because the bug had two costs: the embarrassing UI artifact users were seeing right now, and the architectural debt that produced it.
Same-day: a backend sweeper plus a frontend clamp. The sweeper ran on a 5-minute interval, queried for rows matching the orphan invariant — no terminal outcome, empty body, created more than 60 seconds ago — and force-resolved them to a failed state with the message "Turn interrupted before completion." The clamp turned any non-positive duration into null so we'd stop rendering Done · 0ms while the sweeper caught up. Bandage on the symptom, safety belt for new orphans. Users stopped seeing zombies within an hour.
Week-long: a five-wave rearchitecture. Wave one — the load-bearing change — was a schema migration promoting the outcome column to NOT NULL with an ENUM that included pending as a real first-class state, plus typed columns for fields we'd been smuggling through JSON metadata. NULL stopped being legal. A row in progress was now known to be in progress, not unknown. Wave two wrapped writes in a repository class that enforced the new contract. Wave three rewired the services using the old writer. Wave four updated the frontend hydrate path to use the explicit pending state instead of defaulting nulls to happy paths. Wave five deleted every old code path — not deprecated, deleted — so the old shape became unreachable.
By the end of the week, a Done · 0ms row could not exist in the database. Not because we were vigilant. Because a CHECK constraint refused to write one.
What the screenshot taught us
We wrote down a rule that week: a screenshot of a wrong number in the UI is rarely just a number. Before you clamp a value in the view layer, ask why the underlying state was that value at all. The number is a probe into the schema. If the probe returns something wrong, the schema is what needs fixing, not the probe.
If I'd shipped the ten-line clamp at 10am, the user would have been satisfied, and our turns table would have continued accumulating zombies forever. Six months later: hundreds of thousands of them. The sweeper would have been a multi-hour query. The schema migration would have needed a backfill measured in days. Treating the screenshot as cosmetic would have bought a same-day win and a year-out catastrophe. Triaging it as a class bought a same-week fix and zero recurrence.
The screenshot turned into six follow-on engineering posts: the-0ms-zombie on the bug itself, defense-in-depth-agent-loops on the mitigation stack, typed-columns-over-metadata-json on the schema migration that anchored the rearchitecture, wave-shaped-refactors on how we sequenced the waves, ban-raw-column-references from the repository-class wave, and stop-fixing-bugs-start-fixing-classes as the meta-lesson.
The user got a reply at 5pm. We said the cards were fixed. We did not say what we'd just learned about our database.