Inventory before refactor: a case study where 3 investigators rewrote our plan in 30 minutes

Sibling to investigators-before-builders. Here's the concrete case: three parallel read-only investigators caught that only 8 of 32 fields warranted promotion — and turned a new-table refactor into an extend-table refactor, saving two weeks.

AppX team · 2026-05-15

A sibling post on this blog argues the principle: dispatch read-only investigator agents before builders, every non-trivial task. This is the case study. One specific refactor where three parallel investigators rewrote a two-week plan into a five-day plan in roughly thirty minutes — and where the kind of error the investigators caught is worth naming, because it isn't the kind teams usually look for.

The plan we almost shipped

The trigger was a bug class we'd been chasing for weeks: a class of bug where rows in our agent-state table get stuck mid-flight. The agent finishes (or crashes, or gets cancelled), and the row sits there in an in-progress state forever, because the writer that was supposed to finalize it died on a path we hadn't instrumented. The metadata column was a JSON blob with no schema, so the rot was invisible until something downstream tried to render it and tripped on a missing field.

Diagnosis felt obvious. The metadata blob was the rot site. The blob was untyped. So: build a new dedicated table with proper typed columns, write a migration to backfill from the old table, dual-write for a deprecation window, point readers at the new table, drop the old one.

We sketched the plan. Five engineer-weeks. Two of those were the lift-and-shift and the dual-write window. Estimate felt right. Plan felt right. We were about an hour from cutting the first ticket.

Instead we paused and dispatched investigators.

The investigator dispatch (with three brief shapes)

Three read-only agents, in parallel, each with a single concrete question. Cavecrew-style — minimal tool surface, structured return format, no Edit/Write/mutating Bash. The whole round was kicked off in one message.

Inventory agent #1 — what's actually in the blob today:

investigator:
  question: "List every field name written to the metadata JSON column of agent_state. For each, return: field name, sample value shape (string/number/object/array of X), the writer call site (file:line), and whether anything reads it."
  return_format:
    - field: <name>
      shape: <type>
      writer: <file:line>
      readers: <count + sample paths>
  allowed_tools: [Read, Grep, Glob, Bash(read-only)]
  forbidden_tools: [Edit, Write, NotebookEdit, Bash(mutating)]

Consumer agent #2 — every frontend reader of the metadata fields, with their defensive walk patterns. Same brief shape, scoped to frontend/src/**, return format included component:line, field accessed, fallback chain, what it renders.

Service caller agent #3 — every backend service method that writes to or reads from the existing table. Method name, call sites, whether the call sits on the hot user-facing path or a batch job.

Three briefs. Three concurrent agents. Median wall time, ~9 minutes each, run in parallel. Total dispatch-to-findings: under fifteen.

What the findings actually showed

The aggregated table came back like this:

Eight fields out of thirty-two warranted typed columns. The other twenty-four were either telemetry nobody queried, genuinely free-form payloads where the JSON shape was the right call, or dead fields kept alive by inertia.

The consumer agent surfaced a second finding the inventory agent couldn't have: 17 frontend readers across 8 components were already running four different defensive walk patterns over the same logical field (meta?.title ?? meta?.turnTitle ?? meta?.summary ?? meta?.text). Promoting those to typed columns would collapse four fallback chains into one. That wasn't on the original plan at all.

The service-caller agent surfaced the third one, and this was the load-bearing finding: the existing table already had a partial typed-column overlay. Three columns — turn_title, worked_for_ms, turn_outcome — had been added in an earlier refactor that nobody finished. The migration path we were about to write for "lift everything into a new table" was, almost exactly, the migration path the earlier refactor had started. We'd have spent two weeks doing it again, in parallel, in a different table.

The plan after

The new plan, drafted in the same session as the findings:

ALTER TABLE agent_state
  ADD COLUMN trace_id VARCHAR(64),
  ADD COLUMN plan_summary TEXT,
  ADD COLUMN waiting_ms INT,
  -- ... 16 more typed columns for the 8 promoted fields + their indexable peers
  ADD INDEX idx_trace_id (trace_id);

No new table. No lift-and-shift. No dual-write window. Extend the existing table with nineteen typed columns covering the eight promoted fields plus their indexable peers. Keep the JSON column for the six genuinely free-form shapes. Drop the four dead fields in the same migration. Migrate the writers to fill the typed columns; leave the readers untouched in step one and let them keep reading from JSON via a view-like accessor; cut readers over wave by wave.

Five waves, each independently shippable, each behind its own behavior gate. (The wave shape itself isn't the point of this post — that's the sequel — but the headline is that the work fit in a fraction of the original five-week estimate.)

The bug class the original plan was chasing? Still fixed. The eight typed columns are exactly the fields the rot was happening on. The fix worked. We just didn't need a whole new table to deliver it.

What the investigators didn't catch (honest limits)

Three things the read-only round missed, in the spirit of not pretending this is magic:

1. A real-time path the static call graph didn't surface. One of the "0 readers" fields in the inventory was being read by a WebSocket gateway that pulled it via a dynamic property accessor (row[fieldName] where fieldName was config-driven). Grep didn't find it. We caught it during the first wave when a frontend toast went blank. Lesson: dynamic property access is invisible to read-only investigation. Add a runtime trace pass for any field you're about to drop.
2. Index cost of the new columns. The investigators correctly told us which fields needed columns. They didn't tell us which needed indexes, and nineteen new columns with the wrong index choices would have been an avoidable write-amplification hit. We did a separate query-plan pass before the migration shipped.
3. The migration ordering question. "Can we backfill the new columns from the JSON in a single statement, or do we need an online migration?" was a separate question that needed its own investigator round on table size and lock behavior. We didn't get it for free.

Takeaways

• The original plan was correct about the bug and wrong about the fix. The investigators caught the fix error, not the bug error. That's the load-bearing value: not "is there a bug" but "is the proposed solution shaped correctly for the actual code".
• Three structured-return briefs in parallel, fifteen minutes wall-clock, cost roughly one builder run. The plan shrank from five weeks to one. The ROI math isn't subtle.
• Inventory before refactor is a different question from "find the bug". You're asking the codebase to defend the scope of the proposed change, not the existence of the problem.
• Read-only is the load-bearing constraint. The same brief handed to a builder would have started a partial migration mid-investigation. The investigators returned tables. The planner — with all three in front of it — picked the smaller, correct shape.
• When investigators find that a partial version of your plan was started by someone six months ago and abandoned, finish that one. Don't start a third.

Try your own app idea

Describe your app in AppX →