Defense-in-depth for stateful AI-agent loops: 4 layers, each catching what the others miss

A frontend clamp papers over a bad column. A backend sweeper catches symptoms not causes. A DB constraint blocks bad writes but doesn't repair existing rows. You need all four — and the order to retrofit them when starting from a broken system.

AppX team ·

Defense-in-depth for stateful AI-agent loops: 4 layers, each catching what the others miss

Stateful AI-agent loops produce a category of bug single-layer fixes can't kill. The agent crashes mid-write. A column drifts from the type the code expects. A legacy path bypasses the new repository wrapper. A row gets inserted with durationMs: -1 because some upstream timer never started.

Every time one of these surfaces, there's a temptation to fix it where it showed up — usually the frontend, because that's where the user reported it. Clamp the number and ship. Done.

Except you're not done. You're done with that row. The next bug in the same class slips past the clamp, because the clamp only knew about the symptom you'd seen.

The fix is to stack layers. Each catches a different class of failure. The gaps in one are blocked by the next.

The four layers

flowchart TD
  S["Agent-turn state"] --> L1["Layer 1 · Frontend clamp<br/><i>cosmetic, instant</i>"]
  L1 --> L2["Layer 2 · Backend sweeper<br/><i>catches stuck rows</i>"]
  L2 --> L3["Layer 3 · One-row-per-turn invariant<br/><i>structural</i>"]
  L3 --> L4["Layer 4 · DB CHECK constraint<br/><i>the wall</i>"]
  L4 --> OK["Correct state — guaranteed"]
  style OK fill:#13241a,stroke:#2ecc71,color:#d7f7e3
  style L4 fill:#13203a,stroke:#5e6ad2,color:#dbe2ff
  1. DB layer. CHECK constraints, NOT NULL, ENUMs. The wall the schema enforces at write time. Bad rows can't be inserted, period.
  2. Repository layer. Typed query builder, no raw SQL. Compile-time guards on column names. Drift between code and schema becomes a build error, not a runtime mystery.
  3. Cron sweeper. A periodic read query for invariant violations. The safety belt for rows that got in before the constraint existed, or via routes that bypass the repository — process crashes mid-write, manual SQL during incidents, legacy modules nobody dares touch.
  4. Frontend defensive rendering. Clamps and fallbacks. Handles "stale row, no time to migrate" gracefully. Ships in minutes; covers the gap until backend rollouts land.

Think of them as fence posts. Each one keeps out a different attacker, where "attacker" means "the next bug like the last one." No single fence is enough.

What each layer catches

  • DB constraint catches every write that violates an invariant. If durationMs >= 0 is a CHECK in the schema, no service can insert a negative row, no matter how broken its code is.
  • Repository catches code-schema drift at compile time. Rename duration_ms to worked_for_ms and every call site against the old column fails the type-check before Docker builds.
  • Sweeper catches rows that slipped through historical gaps. The CHECK you added today doesn't retroactively fix the 12,000 rows from last month. The sweeper finds them and repairs or alerts.
  • Frontend clamp catches user-visible regressions between "bug exists" and "all bad rows are gone." A durationMs <= 0 check returning null ships in minutes; the DB migration ships in days.

What each layer misses

  • Frontend clamp misses the underlying corruption. The user sees a sane UI; the row is still wrong; analytics and exports see garbage.
  • Sweeper misses bugs that produce well-formed but semantically wrong data. A row with durationMs: 3600000 looks fine. If the real elapsed time was 36 seconds and someone forgot a divide-by-100, the sweeper has no idea.
  • Repository misses raw-SQL bypasses. Anything via db.execute(rawSql) — migrations, admin scripts, that one legacy module — sails right past the typed builder.
  • DB constraint misses rows that already exist. Adding CHECK (duration_ms >= 0) to a table with 50 negative rows fails the migration. Clean first, then constrain.

The pattern is symmetric: every layer has a hole, and the hole is exactly what the next layer up (or down) is designed to plug.

Concretely

The DB constraint is the cheapest at runtime and the most expensive to add. It looks like this:

ALTER TABLE turn_traces
  ADD CONSTRAINT duration_ms_nonneg CHECK (duration_ms >= 0);

ALTER TABLE turn_traces
  MODIFY COLUMN status ENUM('pending','running','complete','failed') NOT NULL;

You can't add it until every existing row satisfies it. That's the migration cost.

The repository is a one-time refactor, repaid forever. Pick your typed builder — Drizzle, Prisma, Kysely, sqlx — and stop writing raw SQL outside migrations:

async function recordTurn(traceId: string, durationMs: number) {
  return db
    .insert(turnTraces)
    .values({ traceId, durationMs, status: 'complete' });
  // if `duration_ms` gets renamed in the schema, this line fails tsc.
}

The sweeper is a cron that runs a read query, bounded in cost:

@Cron('*/15 * * * *')
async sweepInvariants() {
  const bad = await db
    .select()
    .from(turnTraces)
    .where(or(lt(turnTraces.durationMs, 0), isNull(turnTraces.status)))
    .limit(1000);

  if (bad.length) {
    this.logger.warn(`Found ${bad.length} invariant-violating rows`);
    await this.repair(bad);
  }
}

One query per interval. Cap the limit so a runaway query can't take down the DB.

The frontend clamp is the cheapest by every measure except total coverage:

function DurationLabel({ durationMs }: { durationMs: number }) {
  if (durationMs <= 0 || !Number.isFinite(durationMs)) return null;
  return <span>Worked for {formatMs(durationMs)}</span>;
}

Ten lines. Deploys in minutes. Doesn't fix the data. Buys you time.

The order to retrofit when starting from a broken system

The order to add these layers is the reverse of the order to think about them.

  1. Frontend clamp first. Ten lines, ships in minutes, hides the symptom while you fix the cause. A tourniquet — don't mistake it for a real fix.
  2. Sweeper second. A read-only view of the damage. Run it once, log the count. Twelve rows or twelve thousand? That drives the migration plan.
  3. Repository third. Refactor call sites onto the typed builder. Days of work and a Docker rebuild. Worth it: it stops new bad rows at the source.
  4. DB constraint last. With the sweeper repairing history and the repo stopping new violations, the migration lands cleanly. The constraint is the permanent wall.

Doing it in the other order is how you get a six-hour migration that fails at row 47,000 and rolls back.

When you might skip a layer

  • No legacy data? Skip the sweeper. Greenfield systems with the constraint from day one don't accumulate violations.
  • Server-rendered frontend? Skip the clamp. The server has the constraint; nothing to defend.
  • Brand-new repository with five call sites? Postpone the typed-builder refactor — five sites are grep-able, fifty aren't.

But you cannot skip the DB constraint. It is the only layer that prevents future drift permanently. The other three help you survive the present.

Takeaways

  • Single-layer fixes for state bugs are tourniquets, not cures. They hide the symptom and let the cause keep generating new symptoms.
  • Stack four layers: DB constraint, typed repository, cron sweeper, frontend clamp. Each catches a different class.
  • The order to add them is frontend → sweeper → repo → DB. Cheapest-to-deploy first; permanent-and-expensive last.
  • Every persisted invariant should be checkable at multiple layers. The DB is the only one that prevents bad data from entering; the others fill different gaps.
  • If you've fixed the same bug twice, you're missing a layer. Find out which one.

Try your own app idea

Describe your app in AppX →