The four breakthroughs that turned AppX from demo to product
Smart retry on critical files, intent detection with word boundaries, planning before generation, and credit holds. None was a single algorithm — each was a structural change to the loop the LLM lives inside.
AppX team ·
Eighteen months ago, AppX generated a working mobile app on the first try about 40-50% of the time. That number is enough for a private beta where you've personally apologized to every user. It is not enough to put a signup form on the open internet.
The journey from 40-50% to north of 80% first-try success was not one algorithm. It wasn't a smarter model — the model has changed three times since, and each swap moved the number by less than you'd expect. The breakthroughs were structural. We kept changing the shape of the loop the LLM lives inside until the loop forgave the model's mistakes, recovered cheaply, and stopped pretending unknowns were knowns.
Four changes did most of the work. Here they are, in the order we shipped them.
1. Smart retry on critical files
The first version of our retry policy was binary. If validation failed, we re-ran generation. If it passed, we shipped. The problem: validation passing is not the same as the app being launchable. You can pass a syntactic validator with a colors.ts file that never got generated. The app will crash on the first import, at runtime, on the user's phone.
So we kept generating "valid" apps that didn't run.
The fix was to stop treating every missing file as equal. We added an importance tier: certain files are load-bearing. Route files like _layout.tsx, theme files like colors.ts and theme.ts, the central type file at types/index.ts — if any of these are missing, retry immediately, regardless of how complete the rest of the generation looks. Missing copy, an unused helper component, a placeholder image? Don't burn tokens on a retry; let it ship.
Before: overall coverage threshold, ~40-50% first-try success. After: file-importance gates, ~48-58% first-try success on the same workloads.
Eight percentage points from a single change. The cost was almost nothing — we already had a file list; we just stopped treating it as a flat set.
In hindsight, the deeper lesson is that "validation passed" is a leaky abstraction. A generation pipeline has many layers of "okay," and they don't all mean the same thing. The retry policy was making decisions on the wrong layer.
2. Intent detection that respects word boundaries
Our chat router decided which screen a user wanted to build using string.includes().
You can guess where this is going. A user typed "build a homework tracker." The router saw "home" inside "homework," routed them to the home-screen generator, and produced a beautiful home screen for an app the user never asked for. We have logs of this. They are funny in the way bugs in production are funny.
We rewrote intent matching with word-boundary regex and a scoring function:
const SCORES = {
exact: 1.0, // /\bhomework\b/i
typeBoundary: 0.6, // adjacent to a type token: "homework_screen"
substring: 0.3, // inside another word — last resort
};
const THRESHOLD = 0.5;
An intent now needs to clear 0.5 to fire. A pure substring match (0.3) no longer wins. "Homework" stops matching "home." More importantly, we can add new intents without auditing whether their name happens to be a substring of another common word — every new intent is safe by construction, because the matching rule encodes the actual semantic boundary.
This wasn't a 10-point success-rate jump. It was tail-killing: a long, embarrassing class of "wrong screen generated" bugs that ate hours of debugging and made the AI feel stupid. The fix took an afternoon. The lesson took longer: includes() is a substring check, and substring checks are not text understanding. Anywhere user intent flows through a substring check, you have the same bug shape waiting to happen.
3. Planning before generation
The original generator was one LLM call. Prompt in, multi-file output out, hope. The model held the entire mental model of the app in its head while writing each file in sequence. By file four it had forgotten what file two was named.
We split the loop in two. An architect model (Gemini 3.1 Pro) builds a plan first — a structured JSON describing screens, components, data shapes, and the dependency edges between files. A builder model (the corresponding Flash tier) consumes the plan and generates each file. The architect is expensive and runs once per app. The builder is cheap and runs per file. The plan stays in the prompt as ground truth across every builder call.
The plan looks roughly like this:
{
"screens": [
{ "id": "home", "file": "app/(tabs)/index.tsx" },
{ "id": "detail", "file": "app/detail/[id].tsx" }
],
"files": [
{ "path": "lib/theme.ts", "imports": [] },
{ "path": "lib/storage.ts", "imports": [] },
{ "path": "app/(tabs)/index.tsx",
"imports": ["lib/theme.ts", "lib/storage.ts"] }
]
}
Before generation begins, we run Kahn's topological sort on the import edges. If the sort fails — there's a cycle — we reject the plan and ask the architect to revise. Rejecting a bad plan costs one extra Pro call. Generating an app from a bad plan costs a dozen Flash calls plus a full retry. The cycle-check pays for itself the first time it fires.
The architect/builder split was the single largest structural win: first-try success climbed past 70%, and the average cost per app dropped to around $0.10, because the cheap model now does the bulk of the work. In hindsight we'd have split the loop earlier. Letting one model both plan and execute means it never has to commit to a plan, which means it never has to be self-consistent. The split forces commitment, and commitment forces coherence.
4. Credit holds: reserve, finalize, cancel
The first version of credit accounting was charge-after-the-fact. Generate the app, then deduct credits. On failure, refund. This was simple, wrong, and ugly.
Wrong, because in the window between starting a generation and finishing it, the user's balance was unchanged. Two parallel generations could each see "enough credits" and both proceed when only one fit the budget. Ugly, because on failure the UI showed the deduction landing and then unwinding — balance flickered. The credit ledger filled with paired charge/refund rows that meant "nothing happened." Reading it was like reading a bank statement where every transaction was rolled back.
We rewrote credits as reserve-then-finalize, the way real banks model holds:
start of turn:
holdId = reserveCredits(userId, estimatedCost, ttl=10min)
// balance.available decreases now; balance.total unchanged
on success:
finalizeReservation(holdId, actualCost)
// hold becomes a real charge; balance.total decreases
on failure or cancel:
cancelReservation(holdId)
// hold released; balance.available restored
on process death (never reaches success/cancel):
sweeper finds holds older than ttl and cancels them
The available balance updates the moment a turn starts, so two parallel turns can't both think they have enough credits. The ledger only records real charges. Failures produce zero rows, not paired ones. And the 10-minute auto-expiry is the actual safety net: anything that gets to reserve and never makes it to finalize or cancel — typically because the process died — falls off on its own. The cleanup runs in a fresh process, the way reliable cleanup always has to.
If we'd started here, we'd have saved ourselves a quarter of writing "refund" logic. Reserve/finalize/cancel is the right shape for any operation that has a cost you can estimate at start and only know exactly at end. Charge-then-refund is the wrong shape, even though it feels simpler. The "simplicity" is borrowed against future flicker and a noisy ledger.
What they have in common
None of these four changes is about the model. The model has gotten better, sure, but the largest delta came from changes that would have helped any model — past, present, or future.
- Retry policy stopped treating "validation passed" as "app works."
- Intent routing stopped treating substrings as semantic matches.
- Generation stopped pretending one call could hold a whole app in its head.
- Accounting stopped pretending failures are just successes that get undone.
Each one removed a place where the system was conflating two things that looked the same and weren't. Each fix made the loop more honest about what it knew and what it didn't. The 80% number didn't come from a smarter LLM. It came from a less-credulous loop around the LLM.
That's the thing nobody tells you about shipping AI products. The model is a black box, and the black box has a known error rate. The art is in the scaffolding: how you decide what to retry, how you route work, how you commit to a plan before executing it, and how you account for cost when you don't know the final number yet. Those four problems are not AI problems. They are systems problems, and they predate AI by decades.
We just had to get embarrassed by them, in production, before we admitted it.