Three things we built and threw away this year
The Railover orchestrator fork, the one-shot bundle server, the V1 chat intent classifier. Each was a real working system in production. Each got deleted when the shape of the problem changed underneath it.
AppX team ·
Most engineering blog posts are about the things that worked — the clever fix, the 10x improvement, the architecture that scaled. Those are easy stories to tell, because the ending is already happy.
The instructive stories are the other ones — the systems we built, ran in production for months, and deleted. Not because they were broken, but because the shape of the problem changed underneath them.
This is the autopsy of three. AppX is an AI mobile app builder: users describe an app, AI generates React Native code, a phone renders the preview. Across the year we shipped a lot. We also threw away a lot. These three deletions taught us more than most of the things we kept.
1. The Railover orchestrator fork
Our first container orchestrator was Railover, a fork of a popular open-source PaaS framework. The framework had a sensible model: each app gets a container, the container runs forever, the orchestrator handles deploys and rolling restarts and SSL. We needed container orchestration; the framework existed; the fork started as a few patches.
It worked. We ran it for months. We knew its failure modes intimately.
The framework's mental model was one app, one container, long-lived. Ours was one user, many ephemeral containers, warm pool, dense packing. We needed to pre-warm sandboxes for instant previews, bin-pack a 24GB box with dozens of containers, evict idle ones aggressively. None of these were things the upstream design wanted to do.
Every primitive we touched, we were fighting the framework. The app concept assumed long-lived names; we wanted disposable ones. The deploy pipeline assumed code and container were coupled; we wanted to push code into a pre-warmed container. The state machine had no "warm but unclaimed" node.
We rebuilt it as Forge — a Go control plane and node agent, purpose-built for the pool model. Smaller surface area, state transitions modeled around our lifecycle. The migration took weeks and was painful, but after the cutover the orchestrator stopped being a thing we worked around.
The lesson is unsexy: a fork is a tax. You inherit the upstream's assumptions about the right shape of the problem, and if your shape doesn't match, the tax compounds with every patch. There is a point where rebuilding is cheaper than continuing to patch, and that point arrives earlier than you think.
2. Bundle Server v1
The Bundle Server is what turns a generated React Native project into something a phone can render. Version one was simple: when the code changed, spawn expo export:embed, wait for the bundle, hand it to the device.
It worked. It took roughly seven seconds per platform per update.
Seven seconds is fine if a human edits code, hits save, and waits. Seven seconds is fatal if an AI agent is iterating in a loop — generate, render, see what broke, generate again. Every iteration paid the full cold-start cost. The product felt slow because most of its wall-clock time was a fresh Node process spinning Metro back up from cold.
The root cause was the shape of the workload, not the bundler. expo export:embed is a great tool for shipping a final artifact. We were using it for hot iteration — paying release-build cost on every keystroke.
We rewrote v2 as a persistent Metro dev server, one per container, kept warm across edits. A file write into a running Metro instance is roughly four milliseconds. The incremental bundle is around 100ms. End to end, the same code-push operation that took seven seconds takes a couple of milliseconds — about 1,750x faster than v1. We wrote about the mechanics in a separate post (persistent-metro-1750x-faster-code-push).
The lesson: long-running build processes beat fresh-process-per-request for any workload that iterates. The cold start is the work. The bundler was never the bottleneck; the lifecycle around it was. We threw away a system that worked because the access pattern moved out from under it.
3. The V1 chat intent classifier
When we built the first chat interface, we had a heavy intent classifier in front of the model. Every user message went through a small LLM call that decided: create a screen, edit a screen, change the design system, general question. Roughly 533 LOC of classifier service, prompt, routing table, and dispatch logic.
It worked. The routing was usually right. We shipped on it.
Then we built Edit Engine v2 — a real agent loop with a structured architect tool, where the architect's first job on every turn is to call submit_plan and declare what it intends to do. Inside that plan is the intent: which file, what kind of change, why. The architect was already classifying intent, with more context than the upstream classifier had, because the architect had read the project files.
For a while we ran both. The classifier decided one thing; the architect decided another; occasionally they disagreed — and when they did, the user got worse behavior than either alone, because the classifier had already shaped the prompt the architect saw. Two views of the same fact, drifting, with no merge step.
We deleted the classifier. The architect's tool call became the single source of truth for intent. The 533 LOC went away. The disagreements went away. Latency improved because we removed an LLM hop. Nothing got worse.
The lesson is the one Bloom seems to have figured out too, looking at how their stack is shaped: classification belongs as close to the consumer as possible. A separate classifier upstream is two views of the same thing, and two views of the same thing eventually disagree. If the agent is going to think about the request anyway, don't make something else think about it first.
What these three have in common
None of these systems were broken. Each one shipped, ran in production, and did its job. We deleted them anyway, because:
- The shape of the problem changed. Railover assumed long-lived apps; we needed ephemeral sandboxes. Bundle Server v1 assumed human-scale iteration; AI loops are 100x faster. The classifier assumed shallow routing; the architect made deep planning available.
- The cost of keeping each one upright was rising. Patches against upstream Railover compounded. Cold-start time was becoming the product's worst latency. Classifier disagreements created bugs that were structurally hard to debug.
- The replacement was smaller, not bigger. Forge is less code than our Railover fork's patches. Bundle Server v2 is conceptually simpler than v1. Deleting the classifier removed code with no replacement at all.
That last one is the tell. If your "rewrite" is bigger than what it replaces, you are usually adding scope, not solving a structural problem. If the rewrite is smaller — fewer concepts, fewer moving parts — you found a real misfit between the old design and the actual problem. Those are the rewrites worth doing.
The hardest engineering judgment is not what to build. It is what to throw away, and when. A working system is the most expensive thing to delete, because the system isn't visibly hurting anyone. But every working-but-misshapen system is a tax on every future change. Pay the deletion cost early — your future self will pay it eventually anyway, with interest.