SIGKILL is the boss fight: design your sweeper as your worst-case process-death handler

SIGTERM gives you cleanup time. SIGKILL doesn't. Deploys, OOM, and crashes all skip your graceful-shutdown handler. Your sweeper IS the SIGKILL handler — and its query shape should test for invariant violations, not row age.

AppX team · 2026-05-18

Every "graceful shutdown" handler I have written ran in the cases that did not matter and skipped the cases that did.

A SIGTERM handler is what you reach for when you imagine your process dying — close the DB pool, flush writes, mark in-flight rows as cancelled, release reservations. Tutorials love this. Postmortems do not. Postmortems get written about the cases where that handler never ran.

Three signals. Only one gives you time to clean up.

Three signal classes

flowchart TD
  INT["SIGINT<br/>Ctrl-C, dev"] --> G1["graceful cleanup runs"]
  TERM["SIGTERM<br/>orderly shutdown"] --> G2["a few seconds to clean up"]
  KILL["SIGKILL<br/>deploys · OOM · crash"] --> G3["no cleanup — instant death"]
  G3 --> R["design for THIS case"]
  style KILL fill:#3a1414,stroke:#c0392b,color:#f7d7d7
  style R fill:#13241a,stroke:#2ecc71,color:#d7f7e3

SIGTERM is the polite case. Your handler runs, closes connections, marks the row cancelled. Every blog post optimises for this.

SIGKILL is the rude case. The kernel does not call your handler, your destructors, or your buffered-write flushes. The process is gone between two instructions.

A crash or panic is functionally identical to SIGKILL for recovery purposes. Same failure class.

Why SIGKILL is the realistic case

Every container orchestrator follows the same script: send SIGTERM, wait N seconds, send SIGKILL to whatever's left. Kubernetes defaults to 30s. Docker defaults to 10.

Now think about an AI agent turn. Architect plans for 10-20s, tool loop for another 20-60s, long generation 2-3 minutes. The grace window is shorter than the work. When a deploy lands mid-turn, the math is fixed: SIGTERM, 30s, SIGKILL, dead. Your "mark turn as cancelled" handler does not run — the turn was still inside the model call when the kill arrived.

OOM is worse. The kernel doesn't send SIGTERM. It picks a victim by oom_score, sends SIGKILL, reaps. Your handler is not part of that conversation.

If your state-recovery design depends on cleanup handlers, deploys-during-traffic, OOM-under-load, and panics-in-prod leak rows forever.

Sweepers as the SIGKILL handler

Your sweeper is the only code that runs after the process is dead. That framing makes it stop feeling like a safety belt and start feeling like the primary recovery mechanism it actually is.

A sweeper is a cron job that asks the database: "are there rows that should be in a terminal state and aren't?" If yes: repair them. It runs in a fresh process, independent of the one that died. Nothing it touches has to be in memory.

The sweep interval is the maximum user-visible orphan window. A 5-minute interval = a user sees a zombie for up to 5 minutes. Pick against UX tolerance and query cost.

Inversion of the textbook design: instead of making every process-death path clean, accept that some paths can't be, and put the cleanliness in a separate process that runs unconditionally. Not a handler — a reconciliation loop.

The query shape: invariant, not age

The load-bearing detail. Turns a sweeper from "works most of the time" into "provably correct."

Wrong:

-- DON'T: age-based, matches healthy rows
SELECT id FROM agent_turns
WHERE created_at < NOW() - INTERVAL 24 HOUR;

This matches every old row, including ones that completed successfully years ago. Build the sweeper on this and you'll repair things that did not need repair.

Right:

-- DO: invariant-based, only matches violations
SELECT id FROM agent_turns
WHERE outcome IS NULL
  AND message = ''
  AND started_at < NOW() - INTERVAL 5 MINUTE;

outcome IS NULL AND message = '' is the invariant: no terminal write has landed. The age predicate is a confidence threshold — if the placeholder is older than the max plausible turn duration, the writer is dead by definition. Healthy rows don't match because their outcome is set.

Pick the predicate against the structural invariant that distinguishes "in flight" from "leaked." Credit holds: released_at IS NULL AND captured_at IS NULL. File writes: tmp_path IS NOT NULL AND final_path IS NULL. Age is secondary.

(See the sibling post on the 0ms-zombie for the data-shape angle on this bug class.)

Idempotency requirements

The sweeper will run again before it has finished, sometimes. Two replicas hit the same cron at the same minute. A retry fires before the previous sweep's writes have settled. Design for it.

The UPDATE in the repair step must include the same invariant clause as the SELECT. Re-running must be a no-op on rows the previous sweep already fixed.

// every 30s
const stale = await db.query(`
  SELECT id FROM agent_turns
  WHERE outcome IS NULL
    AND message = ''
    AND started_at < NOW() - INTERVAL 5 MINUTE
`);

for (const row of stale) {
  await db.execute(`
    UPDATE agent_turns
    SET outcome = 'failed',
        error = 'interrupted',
        completed_at = NOW()
    WHERE id = ?
      AND outcome IS NULL          -- still the invariant
      AND message = ''
  `, [row.id]);
}

The WHERE outcome IS NULL on the UPDATE is the load-bearing part. If two sweepers race, one wins, the other's UPDATE matches zero rows. No locks, no coordination — the database refuses to update rows that already have a terminal outcome.

The marking matters too. A dead turn gets outcome='failed', error='interrupted'. A dead credit hold gets cancelled with refund. Swept rows must look distinct from user-cancelled or model-failed ones, or your dashboards will lie about your failure modes.

What sweepers CAN'T fix

The sweeper only sees what reached durable storage. If the writer died before its placeholder INSERT committed, there is no row to find. No record that the work was supposed to happen.

That's the boundary. Sweepers handle "row got written, never finalized." They don't handle "row never got written." For that you need a different layer: an idempotency key on the inbound request so the client can safely retry, a write-ahead log on the producer side, or a two-phase commit across stores.

The sweeper is necessary; it's not sufficient. Most systems need both: idempotency on the front so retries are safe, and a sweeper on the back so partial commits get reconciled.

Takeaways

• Graceful shutdown is the easy case. SIGKILL, OOM, and crashes ship the data corruption.
• The sweeper is your SIGKILL handler — the only code that runs after the process is dead. Treat it as the primary recovery mechanism, not a backup.
• The sweep interval is your worst-case user-visible orphan window. Pick it deliberately.
• Sweeper queries must key off invariant violations, not age. outcome IS NULL says something true; created_at < NOW() - 24h says nothing useful.
• Carry the same invariant into the repair UPDATE. Concurrent sweepers need no coordination then.
• Sweepers can't recover rows that never got written. Pair them with idempotency keys at the entry point.
• If your recovery design only works on the SIGTERM path, you don't have a recovery design. You have a happy-path with a fancy name.

Try your own app idea

Describe your app in AppX →