A deploy cost us a user. Here is the 68-minute bug.

A new user tried to build his app three times on a Sunday evening and nothing happened. The cause was my own deploy, and a drain loop that waited 68 minutes for a turn that was only waiting for a human.

Ilyos Olimov · October 20, 2026 · 4 min read

On Sunday, October 19, a new user signed up on his iPhone at 17:22 UTC and typed one of the better prompts I have seen on AppX: a mobile-first AI assistant with a glowing orb, a chat, a memory section for notes and goals, Uzbek as the main language. He clicked Build at 17:25.

Nothing happened.

He created the project again at 17:44. Nothing. Again at 17:46. Nothing. Then he left. All 100 of his free credits were still there the next morning, and three identical empty projects sat in his account.

I only noticed because I look at every signup. When I opened his account in the admin panel the row said: 3 projects, 100.00 credits. That combination should never exist.

What he saw

The screen said "Planning your app..." and stayed there. A small toast appeared for a few seconds: "Server is restarting, please retry in a moment." On a phone you miss that easily. So he did the natural thing, went back and started over. Twice.

What actually happened

At 17:18 UTC I had started a deploy. AppX deploys the API by building a new container, then asking the old one to stop. The old one does not stop immediately. It "drains": it refuses any new build and waits for the builds already running to finish, so that nobody's app dies mid-generation. That part is right. A deploy must never kill a running build.

The problem was what the drain counted as "running".

One user had started a build that morning and the AI had asked him a clarifying question. He never answered. His build was sitting in a state called awaiting_user. Nothing was running. No model was generating. The turn was simply waiting for a human who had closed the tab.

The drain saw that turn as live, because its heartbeat was still ticking, and waited for it. It waited the full cap: 4,109,000 milliseconds. Sixty-eight minutes. During those 68 minutes the old server was still serving the site, still letting people sign up and create projects, and refusing every single build start.

The new server came up at 18:28. By then the user was gone.

Why the cap was 68 minutes

The number is not random. A first build can take two generation attempts plus validation plus the web export. The drain budget is the worst case of all of that plus headroom, so that an honest long build is never cut off. For a build that is really running, 68 minutes is the right ceiling. For a build that is waiting for a human, it is 68 minutes of nothing.

The code even had a rule for this: a turn paused on a question is excluded from the drain. But that rule only recognised the old way of storing a pause, as a chat row of type ask_user. The newer lifecycle stores the question inside the turn's checkpoint, and the turn keeps a heartbeat while it waits. So the rule said "not paused, still alive" and the drain waited.

The same thing had happened the night before, on the previous deploy. I had noticed that deploy took 70 minutes and written it down as a follow-up. I did not connect it to users until I saw his row.

The fix

Two changes, both live since October 20.

On the server, the drain now checks the durable state of the turn first. If the database says awaiting_user or awaiting_plan, the turn is excluded, whatever its heartbeat says, and it is never force-failed at the deadline. The deploy that shipped this fix took two minutes end to end.

In the builder, a refused first build no longer freezes on "Planning your app...". It shows "The server is updating. Your build will start automatically." and when the socket reconnects to the new server it starts the build by itself, once, only if nothing was started in the meantime.

What I did for him

I built his app from his own prompt in my account, tested it, moved it into his account, and emailed him in Uzbek: this was our fault, here is your app, your credits are untouched. Whether he comes back is up to him.

What I took from it

The expensive part of this bug was not the code. It was that a deploy looked successful. Health checks were green. The new container came up. Nothing in the deploy output said "you just refused builds for an hour". I now check for open clarifying questions before every deploy, and the drain logs exactly which turns it excludes and why.

If you build alone, the lesson is simple: every "safe" waiting loop needs a reason to stop that does not depend on the thing it is waiting for.

Ready to try your idea?

Start with a clear description. AppX will help you turn it into an app you can test.

Start describing your app →

← Back to Blog