Full portfolio

isopod — Not an Agent That Writes Code, but a Harness System That Runs an Organization's Development Process

Written 2026-08. The company is named; product and personal names are anonymized. Every number in this document is measured; reproduction commands are in §10.

1. Executive Summary

isopod turns natural-language spec documents into work cards on a dependency graph, then runs multiple agents unattended until that graph is drained, producing verified commits. It was operated for roughly two months on the conversational AI assistant of Cupix, a B2B 3D digital-twin SaaS company.

The measured results:

The agent pipeline executed 73 work cards and got 70 through verification (95.9%). Eight cards a human implemented directly are excluded from this figure.
385 agent passes ran in the process, with zero human approvals in the execution layer.
Measured cost was $2,754, agent compute 144.4 hours, peak concurrency 13 passes.
The SSOT under management was 200 cards / 438 dependency edges.

How This Differs From "Just Telling an Agent to Build It"

A coding agent optimizes for one unit of work. An organization has to run a process that keeps going. The five things real work cares about below are axes most agent harnesses do not address at all, and nothing in the survey (§5, six adjacent tools) did all of them on a single dependency graph at once.

What real work demands	A typical coding agent / SDD tool	isopod	Evidence
Pin the spec rigorously, then prove the implementation conforms	The spec is a prose prompt; conformance is stood in for by "the tests are green"	The spec schema requires a state machine (states, events, transitions — illegal transitions enumerated too) → every acceptance criterion ships falsifying examples → tests target transitions → all four gate layers are adjudicated by deterministic programs	§3-4, §4-2
Report progress at any moment	A chat scrollback is the only state	200 cards as version-controlled SSOT, a 7-state machine, tracker mirroring. "What is where" exists continuously in both the files and the tracker	§3-3, §4-3
Collaborate with other teams	Absent (built on the premise of working alone)	External-team cards are a read-only mirror — ownership is first-class in the data model, write scope is restricted field by field, and the code raises an exception to hold it. What we expect back is pinned in `consumer_contract`	§4-3
Spans that must wait on a human policy decision	Either absent, or so many approval gates that the pipeline runs at human speed	Gates are separated by layer — humans only in the requirements layer, zero in execution. Undecided policy lives as a gate card that is a graph node, so exactly its dependents wait	§3-2, §4-6, §7-2
Deterministic tracking of specs that change mid-flight	Rewrite the prompt — the previous state is gone	Per-state editability rules plus migration cards (a shipped card is never edited; a new path forward is created). Reverse cleanup on retirement, forward injection on new decisions, edge review on supersession — all three codified as audits	§4-3, §7-4
All of the above in one system	Of six adjacent tools: one has built-in tracker integration; none has a multi-team ownership model	The five above operate on one dependency graph, and agents drain that graph on top of it	§5

The left column is this document's thesis: isopod's differentiator is not "the agent is smarter" but that everything real work actually requires around an agent is modeled as first-class.

What Kind of Problem This Harness Was Built For

More important than those numbers is the conditions it ran under. These were not laboratory conditions.

Three teams collaborating through an issue tracker (two internal, one external) had their work in a single graph. The external team's tracker is the source of truth for their cards, so those had to be a read-only mirror — and that ownership boundary is not a documentation convention: the code raises an exception.
The spec kept moving. 20.5% of cards (41) were cut from scope, and 17 architecture decisions accumulated and superseded one another. So the core of this harness is not "how to create cards" but "how to move an already-shipped card to a new destination without falsifying history."
As a result the central design question became "where does policy judgment (LLM) end and deterministic machinery begin?" Where that line was drawn, and how it is held in code, is the subject of §4.

Of six adjacent tools surveyed, only one has built-in tracker integration, and none has a multi-team ownership model (§5). That axis is what isopod gained from real operating conditions.

Limitations Stated Up Front

done now receives runtime truth from a post-merge smoke test, but a failing smoke does not block the state transition (§9-1). Gate exemptions are counted, not blocked (§9-2). Portability was demonstrated once but its economics were not (§9-3). The cost of building the harness and the harness-versus-model attribution were not measured (§9-4). The sample is a single project.

Also: the 95.9% this document cites was obtained before the verification defects in §9-2 were fixed. Those fixes and the exemption instrumentation secure what comes next; they do not retroactively strengthen the 70 cards already counted. The document keeps that distinction throughout.

2. Problem Statement — Why This Was Built

2-1. The Gap Between Spec and Implementation

The failure mode where coding-agent output "compiles but subtly misses the intent" is one the industry converged on naming after 2025. The standard industrial response has been spec-driven development (SDD) — promoting the natural-language spec from a disposable plan to an SSOT artifact that governs implementation.

But SDD tooling has recorded a problem it has not solved: managing specs at organizational scale. Concretely: (a) how to adjudicate which side is true when spec and code diverge, and (b) what convention preserves history when an already-shipped spec has to change.

That is exactly where isopod aimed. The target project already had spec documents; an external collaborating team (the algorithms team) managed its own work in a separate tracker; and accumulating architecture decisions kept invalidating the premises of cards that had already shipped.

2-2. The Trust Problem in Agent Autonomy

The second motivation is how much can actually be delegated. More autonomy raises throughput, but the gap between "what the agent said it finished" and "what actually works" becomes risk directly.

Industry measurement shows the scale of this. In a study classifying 1,642 multi-agent failure traces, the two largest failure modes were step repetition at 15.7% and unaware of termination conditions at 12.4% — together, over a quarter of observed failures are "not knowing when to stop." Premature termination at 6.20% coexists with them, which means a single iteration cap provably cannot address both directions.

So isopod's design question was never "how do we make the agent smarter." It was "what counts as evidence, when do we stop, and where do humans belong?"

2-3. And These Conditions Arrive Together

Either problem above can be partly handled by a single tool. What makes real work hard is that five conditions hold simultaneously. On the target project they genuinely overlapped.

An explicit spec exists, and conformance to it must be provable. Spec documents were already there, and the requirement was not "the agent did well" but a traceable answer to which acceptance criterion was covered by which test.
Progress must be answerable at any moment. The tracker is the surface collaboration happens on, so "what is where" has to be queryable state, not a chat log.
Another team's work sits inside our dependency graph. But their truth lives in their tracker — making them nodes makes them editable, and editing one falsifies another team's state.
Some cards cannot proceed before a human policy decision. Committing code on top of an undecided policy wastes all of it the moment the decision flips. At the same time, waiting must not stop the whole pipeline.
The spec changes mid-flight. 20.5% of cards (41) were cut from scope, and 17 decisions accumulated and superseded one another.

Remove any one and the design gets dramatically simpler. Without (3) the entire ownership model is unnecessary; without (5) neither per-state editability rules nor migration cards need to exist; without (4) there is no reason to split gates by layer. Most of isopod's complexity comes from all five holding at once — a condition a tool for an individual developer never encounters (§5).

3. System Architecture

3-1. Separating Harness From Project

Harness code and project data are separate, and the repository where code actually lands is a third place. The harness points at the target repository; it does not live inside it.

flowchart LR
    subgraph H["isopod harness (the portable part)"]
        Core["core/<br/>runtime drivers · orchestrator"]
        Mod["modules/<br/>SSOT tools · adapters"]
        Sh["shared/<br/>common primitives"]
        Skills["skills/<br/>slash-command surface"]
    end
    subgraph P["Project instance (project-owned)"]
        Cfg["config file<br/>paths · tracker · teams · thresholds"]
        SSOT["200 SSOT cards<br/>+ dependency graph"]
        Spec["spec documents<br/>(human-owned, read-only)"]
        RT["runtime state<br/>(regenerable)"]
    end
    subgraph T["External"]
        Target["target repository<br/>(write target)"]
        Tracker[("tracker<br/>Jira / noop")]
        Ext["external team repo<br/>(read-only)"]
    end
    Cfg --> Mod
    SSOT --> Mod
    Mod -->|"run"| Target
    Mod -->|"mirror"| Tracker
    Ext -.read-only.- Mod
    Core -->|"runAgent()"| Backend["agent backend adapter"]
    Backend -->|spawn| CLI[("agent CLI")]

3-2. Three Nested Loops

The pipeline is not linear. observe → decide → act → verify runs on three different time scales, and the outer loop's "act" is draining the inner loop.

flowchart TB
    subgraph L2["L2 · requirements loop (days–weeks) — human gates live here"]
      direction LR
      O2["observe<br/>spec change · bug · scope cut"] --> D2["decide<br/>anchor · draft cards"]
      D2 --> A2["act<br/>publish cards → frontier"]
      A2 --> V2["verify<br/>observation gate (human)"]
      V2 --> O2
    end
    subgraph L1["L1 · frontier loop (hours–days) — unattended"]
      direction LR
      O1["observe<br/>compute runnable set"] --> D1["decide<br/>execution order"]
      D1 --> A1["act<br/>sweep cycle"]
      A1 --> V1["verify<br/>merge-verify"]
      V1 --> O1
    end
    subgraph L0["L0 · story loop (minutes–hours) — unattended"]
      direction LR
      O0["observe<br/>reality snapshot"] --> D0["decide<br/>spec agent"]
      D0 --> A0["act<br/>implement"]
      A0 --> V0["verify<br/>4-layer gate · review"]
      V0 --> O0
    end
    A2 -.delegates.-> O1
    A1 -.delegates.-> O0
    V1 -.reports drain.-> V2

Humans issue exactly 8 commands: anchor → story:go → (review) → story:jump → roll → jump → (PR review and merge) → land → gate-release. There is not a single approval gate inside L0 or L1.

3-3. Card State Machine

Each work card is one version-controlled file. Editability rules differ per state.

stateDiagram-v2
    [*] --> drafted
    drafted --> planned: review passed
    planned --> published: published to tracker
    published --> done: merge-verify passed
    published --> deleted: scope cut
    planned --> deleted: scope cut
    drafted --> deleted: scope cut
    done --> [*]
    deleted --> [*]
    note right of drafted
      freely editable
    end note
    note right of published
      conditionally editable
      (intent-strengthening only,
       re-publish required)
    end note
    note right of done
      no edits (by convention)
      shipped snapshot
    end note

When a shipped card's destination changes, the original is not rewritten. A migration card is created and linked by dependency — the prior path is preserved rather than overwritten.

3-4. The Verification Gate Stack

A story passes through five personas sequentially. Each persona has different read permissions.

flowchart LR
    R["reality snapshot"] --> S["spec<br/>define acceptance criteria"]
    S --> P["plan<br/>decompose work"]
    P --> T["test-author<br/>write failing tests first"]
    T --> I["impl<br/>implement (cannot edit tests)"]
    I --> G{"4-layer gate"}
    G -->|pass| RV["review + security"]
    G -->|fail| I
    RV -->|pass| C["commit (N→1 squash)"]
    RV -->|reject| I
    C --> PR["PR (after composed-branch re-verification)"]
    PR --> M["human review · merge"]
    M --> LD["land: merge-verify → state transition → tracker sync"]

Every judge in the four-layer gate is a deterministic program.

Layer	What it checks	Judge
L1 static skeleton	declared-but-unused imports, constant-returning stubs	TypeScript compiler API
L2 mutation	whether the tests actually catch defects	AST mutation + test runner
L3 golden I/O	whether I/O is preserved across a port	JSON deep-diff
L4 runtime acceptance	whether real scenarios run	actual process spawn

Red-baseline sits in front of all of it: before implementation, a test must be proven to fail. The proof is the exit code of a subprocess the runner ran itself, not the agent's statement — the test must fail (exit ≠ 0) for red-baseline to be recorded as passing.

4. Technical Highlights

Fifteen industry paradigms were surveyed and mapped onto isopod's implementation. Seven of the most discriminating are covered here. The "where this must not be overstated" note closing each section is a refutation that came out of the investigation, and none of them have been removed.

4-0. The Center of the Design — Where the LLM Is Allowed and Where It Is Not

Every design decision in this harness converges on one question: how much of this is policy judgment, and where does machine work begin?

LLMs are non-deterministic. They phrase the same answer differently on the next call, and sentence shapes shift with minor model revisions. So isopod uses the LLM only for policy judgment, and does not accept even that output as free text.

Stage	LLM used?	Output form
Anchor extraction	two models, independently	structured; cross-checked, symmetric difference must be zero
Anchor gates (single-concern, grouping, retirement, per-kind)	no	deterministic checks
Card drafting (grouping, alignment, triage)	the only policy-judgment point	all four judgments as enum / bool / id
Card publication · cleanup	none (zero model calls)	pure determinism
Runnable-set computation	no	graph operation
Spec · plan · test · implement · review	yes	schema-forced, re-prompted on validation failure
4-layer gate adjudication	none	TS compiler / AST mutation / deep-diff / process spawn
State transition (merge-verify)	no	git ancestry check

The way to read that table is simple — the "no" cells are the work. Handing an agent what agents are good at is not hard. The hard part is picking out what must not be handed to an agent, moving it to a deterministic machine, and holding that boundary in code.

4-1. Determinism at the Anchor Graph — Two-Model Cross-Check and a Content-Based Join

The problem. Extracting units of work from a natural-language spec is inherently the LLM's job. But that output becomes the foundation for everything downstream — get the anchors wrong and the cards, the dependency graph, and the tests are all wrong. A non-deterministic step sits at the very top of the pipeline, and that is the fundamental difficulty of this design.

The approach — require agreement between two different models. Two distinct models, named in configuration, read the same spec and extract an anchor graph independently. The pass condition is not a loose similarity score: the symmetric difference of anchors must be zero AND the symmetric difference of edges must be zero. One mismatch fails the run.

The real design work is in the join key. If two models extract the same content but assign different IDs, a naive comparison reports total disagreement. So the join key is not an ID the model invents — it is a character interval (a span) over the source text. Every anchor must quote the source string it derives from without changing a character (paraphrasing is prohibited), and that quote is a pointer, not an identity: it is resolved to an exact substring within the line window of its citation, converted to a character interval, and the join runs over the intervals.

Intervals were chosen because they solve what string comparison cannot. Two extractors segment a spec independently, so their quotes rarely land on the same characters even when they mean the same claim — one quotes the whole sentence, the other a noun phrase inside it. Under a string join those are total mismatches. Under an interval join, overlap means the same claim, and overlap admits no threshold: two spans share a character or they do not. Relations are decided with Allen's thirteen interval relations, where crossing is forbidden and nesting is allowed (two claims on one line each occupy their own sub-span; a noun-phrase anchor sits inside the constraint anchor that encloses it). The join runs in two phases — same-kind first, then whatever is left across kinds. A single phase lets a small nested anchor drag its enclosing anchor into the other run's component and report a match as a granularity dispute.

Everything after agreement is deterministic. Once the solidness gate passes, adoption — moving the agreed candidate store into the anchor store — runs as pure code, followed by the deterministic checks (single-concern, grouping, retirement, per-kind). From there on, exactly one LLM policy-judgment point remains — card drafting — and even there all four judgments are forced into enum/bool/id fields. Card publication and cleanup make zero model calls, deliberately: irreversible actions (publishing to the tracker) are not placed behind a non-deterministic step.

Adoption becoming code is this design principle applied to itself in hindsight. The step used to ask an agent in prose to "copy the candidate store into the anchor store, renumbering into the configured prefix sequence." But the configuration lists five prefixes and exactly one has kind: anchor — so the set was already known to the code, and the model was asked to reconstruct which member applied, the very shape this harness forbids in §7-3. On one run it duly picked story prefixes by anchor kind, producing a store whose IDs collide head-on with card IDs, and schema, gates and npm run check all passed it. Copy-and-renumber is now code (reference rewriting walks all thirteen sites derived from the schema), and the validator rejects any anchor ID outside the anchor prefix. This change took the anchor pipeline's agent calls from five to four.

Where this must not be overstated. Two-model agreement can miss the same thing twice (correlated error). A zero symmetric difference proves the two agreed, not the spec was read correctly. And this gate exists only at anchor extraction — the spec and implementation personas downstream are single-model.

4-2. Tests Defined by a State Machine — TDD in Action

The problem. Tell an agent to "write good tests" and it writes tests it can pass. Coverage is no defense — measured suites exist with 100% line coverage and a 4% mutation score, and one large-company report found that judging by line coverage alone would have discarded 277 valid tests.

The approach — force the spec to be a state machine, not prose. The spec document's schema makes states, events, and transitions required fields. And each transition carries a legal flag, so illegal transitions are enumerated explicitly too.

Field	What it forces
`transitions[]`	`{from, event, to, legal}` required — illegal transitions are first-class
`acceptance_criteria_coverage[]`	per criterion: `covered_by` + `falsifying_examples` (examples that would refute it)
`test_plan[]`	`{id, targets, type, name}` — each test states what it aims at

A real card from the project: 15 of its 25 transitions were illegal transitions, and against 6 acceptance criteria the test-authoring agent wrote 20 guards before any implementation existed. In other words the majority of the tests are negative tests that prove "this must not happen." Because coverage is measured against transitions rather than lines, there is no path where skimming the happy path earns a green build.

Three layers of enforcement stack on top of that.

Red-baseline — the test must be proven to fail before implementation. The proof is the exit code of a subprocess the runner ran itself, not the agent's claim.
Test/implementation separation — the implementation persona cannot edit test files. The "make it green by editing the test" path is closed by role separation, not by asking a human to watch for it.
The constant-stub unprovability rule — the spec validator rejects a spec whose happy-path and edge-case expectations are identical, because a constant-returning stub must not be able to satisfy both. This is a gate at spec-authoring time, not at implementation time.

Where this must not be overstated. Test/implementation separation is enforced by a post-hoc git diff check, not file permissions. The defect where that check looked at a hardcoded path prefix — letting edits to test files colocated in the source directory slip through — was fixed by referencing the configured test glob and pinned with a boundary test (§9-2), but the check remaining post-hoc is unchanged.

That glob fix then exposed two more defects hiding behind it, both of which surfaced only once a greenfield spec was actually driven through the pipeline (§9-3). (i) The check compared against the roll base, but the tests the test-author writes are laid into the implementation worktree as an uncommitted patch. So tests the implementation never touched were in every diff, and under a TDD flow this gate could never pass — it was paralysis, not enforcement. (ii) git diff does not list untracked files, and a file the implementation just created is untracked. The check therefore caught test files the implementation edited but never ones it authored — the easier of the two evasions. Both are now closed: (i) the comparison is against the content of the test-author worktree (content, not path, so rewriting a handed-over test is still caught) and (ii) untracked files are unioned into the changed set. At the time this document claimed the path was "closed by role separation," that enforcement was not actually standing — because the claim had never been checked by execution, which is precisely the recurrence of the "zero tests crossing the boundary" failure §7-1 names. Also, who fills in the state machine? The spec persona — an LLM. A transition it omits is never a test target in the first place. What this structure enforces is not "did you enumerate exhaustively" but "whatever you enumerated necessarily becomes a test."

4-3. An Ownership Model for Multi-Team, Multi-Tracker Environments

The problem. The external team does not use our SSOT. Their truth lives in their tracker. Yet computing a dependency graph requires their work to exist as nodes in our graph. That creates a dilemma: making them nodes makes them editable, and the moment we edit one we are falsifying another team's state. In real tracker-based collaboration this problem cannot be avoided.

The approach — make ownership a first-class part of the data model.

Branch on capability flags, not team names. The code never decides "this team is external" by matching a team-name string; it branches on capability flags in configuration (does this team own its tracker? can it write to the repo? which schema variant? is it a render target?). Teams can be added or renamed without touching code.
Restrict the write scope field by field. On an external team's card we may write only tracker mirror fields, status, and deadline milestone. The entire body — title, subtasks, dependencies, contracts, anchors — is locked. Creating new cards is blocked too.
This boundary is held by code raising an exception, not by a documentation convention. An invariant check inside the edit function rejects patches outside the mirror scope — verified by actually executing it during the investigation (body edit → exception, mirror field → allowed).
In the other direction, we record the contract. The interface we require from the external team is pinned into the card as a consumer_contract field, and of the 200 cards in the snapshot, all 24 that carry this field are external-team cards. We cannot edit their implementation, but what we expect of it is preserved under version control.

The data model also absorbs the fact that specs change. On this project 20.5% of cards were retired and 17 decisions accumulated and superseded one another. Three rules came out of that: reverse-dependency cleanup when a card is retired, forward-dependency injection when a new decision is published, and dependency-edge review when a decision is superseded. The first two are codified as audit functions; the third came out of the incident in §7-4. When a shipped card's destination changes, the original is left alone and a migration card is created and linked by dependency.

Why this is a differentiator. Of six adjacent tools surveyed, one has built-in external tracker integration and none has a multi-team ownership model (§5). This is less a technical advantage than a difference in problem framing — a tool for an individual developer never encounters this.

Where this must not be overstated.

The external-team mirror is one-way. We poll their tracker and reflect it; it is not bidirectional sync.
Ownership enforcement has an escape hatch — a single environment variable lets a write through, and the only trace is one line on stderr, which lands in neither a file nor a commit. The post-hoc audit is warning-tier and does not block CI.
The prohibition on editing done cards is a convention, not code enforcement (§9-2). What is enforced is the eligibility to enter done (merge-verify), not immutability once there.

4-4. Verification-Driven Loop — Rejecting Self-Attestation at the Shape Level

Industry difficulty: the adequacy of the verifier itself is unsolved. Measured suites exist with 100% coverage and a 4% mutation score, and one large-company case reported that line coverage alone would have wrongly discarded 277 tests. "The tests are green" is not a safety signal.

isopod's implementation — six places that turned out stronger than expected:

Self-attestation is rejected. The evidence-object validator blocks command: "self-review …" as a hard error. Word-boundary anchoring means real tool names (self-review-runner) do not false-trip.
Code overrides the model's verdict. Even when a reviewer asserts ok: true, the code flips it to ok: false if fail-severity findings exist, and it does not honor the escape valve the model claims.
Schema enums are injected from code at load time. Runner-state, failure-category, and reviewer-kind enums are pushed into the schema from the code SSOT, making schema/code drift structurally impossible.
The constant-stub unprovability rule rejects specs whose happy and edge expectations coincide, since a constant-returning stub must not satisfy both.
Merge-verify refuses conservatively when it cannot decide, and re-checks once more immediately before the state transition to catch force-pushes. If unverified, it exits without mutating anything.
Composed-branch re-verification before the PR. This is the only place that catches cross-story breakage the per-story isolated worktree gates cannot see in principle; on failure it does not open the PR at all.

Where this must not be overstated.

The evidence-completeness hole is closed — but timing matters. The result-document validator did not pass the rigorous flag through to the verification function, so a result with zero L1–L4 evidence passed the validation CLI with exit 0 (measured). It was closed by passing the flag through plus a boundary test that spawns the CLI as a process and asserts exit 2 on a zero-evidence document (a unit test cannot catch this class in principle — tests calling the function directly already existed and were all green). But the 70 cards in this document were verified before that fix.
The waiver surface is still wide — it is now counted. N/A determinations for L3/L4, warning-downgrade at L1, and lowering the L2 threshold can all still be turned on by the spec/plan agents themselves, and the stated justification is effectively unvalidated (only checked for non-emptiness). What changed is that every waiver now passes through a single seam into a ledger, with per-surface counts surfaced in the report (§4-8). Why count rather than block: §7-2 — blocking ties the pipeline to human response time.
Therefore the statement "every story passed all four verification layers" is still false. The accurate statement is "each story passed the gate set that applied to it, and from here on what was missing from that set is recorded as data."

4-5. Termination Conditions — Measured Against the Industry's Largest Failure Class

Industry difficulty: as cited above, "not knowing when to stop" is the largest failure mode, and it coexists with premature termination, so a single knob cannot fix both.

isopod's implementation: stopping rules are split into three layers — an outer restart cap (5) with 6 exit paths, two kinds of stall detection that terminalize a story that keeps failing in the same category, and three tiers of retry caps.

Measured result: of 352 (story × persona) pairs, 319 finished in a single pass and 33 took two. All 379 observed termination reasons were normal completion. No runaway repetition loop was observed in this dataset.

Where this must not be overstated: the sample is 385 passes on one project, differing in scale and diversity from the industry study cited (1,642 traces across 7 frameworks). Also, the statement "retries are capped at 10 total" is wrong — three separate budgets (transport errors, gate-harness errors, same-call retries) do not decrement the cap. The accurate statement is "finitely bounded, but the bound is not 10."

4-6. Human-in-the-Loop Placement — A Principle Where the Industry Has None

Industry difficulty: the survey conclusion was that "there is no principle for which layer makes an approval gate useful versus noise, so practice tends to either put them everywhere or nowhere." The adjacent-tool survey reproduces exactly that — one tool has many approval points (everywhere), another specifies no blocking approval point (nowhere).

isopod's answer: split the gates by layer.

Layer	Human gate	Why
L2 requirements	yes	publishing cards is irreversible, and the observation gate is the only channel through which integration-scenario runtime truth arrives (story-level truth now arrives automatically via the post-merge smoke — §7-1)
L1 frontier	none	automatic restart until drained
L0 story execution	zero	385 passes measured, unattended

Where this must not be overstated: some of the five downstream defenses that justify removing execution-layer gates can be waived by the agent, as shown in §4-4. "We removed the gate but other gates catch everything" is only partly true.

4-7. Structured Output — Driving Free Text Out of Control Flow

Industry difficulty: syntactic guarantees do not imply semantic correctness (vendor documentation says so itself). Beyond that, research shows format constraints degrade reasoning, and a follow-up study reported that most of the loss comes not from decoder constraints but from the instruction asking for a format at all.

isopod's implementation: schema validation lives not on a vendor flag but on in-house validation over a normalized result envelope, plus re-prompting. On violation the concrete violation text is appended to the prompt and the agent is respawned (2 retries by default, 3 attempts total); when exhausted the value is emptied. The backend interface documentation pins vendor schema flags as "hints only," so every CLI with an adapter travels the same validation path. Envelope unwrapping likewise uses a quote-aware balanced brace parser rather than a regex.

Where this must not be overstated: the principle (an internal rule: "no regex parsing of free text") is not fully observed. The transport-error classifier does apply a regex to subagent stdout, and reviewer-verdict normalization has one regex without schema validation. The former, however, blocks the prose path whenever a complete event stream is available. The accurate statement is "when a structured signal exists it never looks at prose; only in its absence does it fall back to conservative patterns."

4-8. Self-Improvement — A Demonstration of Partial Mitigation

Industry difficulty: "as rule files accumulate, there is no way to observe which rule actually fired, and no automated means of detecting conflicts or fossilization between rules — a self-improving loop codifies wrong lessons just as effectively."

isopod's partial answer: some rules were promoted into executable audit functions.

Rule	Promoted audit	What it catches
Reverse-dependency cleanup on retirement	dangling-dependency audit	a live card depending on a retired one
Forward-dependency for new decision cards	orphaned-decision audit	a published decision with zero inbound dependencies
Cap on unverified card accumulation	unverified-gate audit	cards behind an observation gate exceeding the configured threshold

There are 22 dependency-graph audits in total, and each catches something schema validation cannot in principle (cross-file uniqueness / referential integrity / graph reachability / filesystem existence / diff against git HEAD / comparison against config).

A second axis of observability — the exemption ledger. This is the direct answer to the first half of the industry difficulty above ("no way to observe which rule actually fired"). The verification stack's ten exemption surfaces each had a legitimate justification, but nobody counted the total — there was zero data on which defense line was nominal and which was real. The fix is not to enumerate the exemption points and tally them (a new point added later silently drops out of the tally — the same trap §4-7 forbids, where no closed complete set can exist). Instead, exemptions pass through a single seam. Every one is appended to a ledger as {roll, story, pass, surface, check, reason_kind}, with every branch-carrying field an enum or id and free text confined to one human-readable slot. Per-surface counts appear in the run report, and a static audit detects any new exemption point that bypasses the seam.

The design principle is "don't block it — count it and surface it." Blocking would directly reproduce the §7-2 incident (a pipeline stalled waiting on a human); once the exemption rate is a metric, pressure appears on its own while the pipeline keeps moving. Note though that this instrumentation accumulates from here on — the 70 cards in this document predate the ledger, so their exemption distribution cannot be known retroactively.

Three ways this must not be overstated.

Only a minority of the 29 rules were promoted to audits; 20 remain prose.
You cannot write "the audits caught N bugs." In the 200-card snapshot exactly one audit actually fires. The honest statement is "past incidents are codified as audit functions, and the physical evidence of that incident is still in the data."
Do not count the 22 as homogeneous — they split into error and warn tiers with warn in the majority, so fewer than half actually turn CI red under default settings.

And this axis contains a self-refutation. The investigation found three cases where the rule documents disagree with the code (a constant the rules describe is absent from production code, a persona-count error in a header, a hardcoded path violating the config-as-SoT principle). That is direct evidence that the self-improving loop fails to prevent drift in its own documentation — a restatement of the industry difficulty above.

The same pattern surfaced once more. The harness contains a sweep tool built to target precisely the boundary-defect class §7-1 prescribes — and a repository-wide grep found the only references to it were the file itself and a single comment. It was built and never called. Then three of the four measured defects in §9-2 turned out to be exactly the class that tool was aimed at. The conclusion of this axis is that building a tool and making it fire are separate pieces of work; the correction was to run that sweep against the harness itself once. What the self-improvement loop actually reaches is not "promote rules into audits" but "is the promoted audit wired into a place that actually calls it."

5. Design Positioning — What Differs From Adjacent Harnesses, and What Does Not

Six adjacent tools were surveyed from primary sources: two SDD toolkits, one agent skill plugin, one autonomous SWE agent, one orchestration framework, and one commercial spec-driven IDE. All observations date to 2026-08-06, and this field moves week to week, so some judgments below will be wrong with time. Read this as a difference in what each tool optimizes for, not as a ranking.

Axis	isopod	Adjacent tools
What gets validated	the work product, by actually executing it	the spec document's form (one enforces programmatically via exit code, one via blocking hooks, two advisory)
Unit of work	a set of cards on a dependency graph	one feature / change / plan / ticket
Scope of one invocation	automatic restart until the frontier drains	one item (multi-item auto-drain unconfirmed for all six)
State lifetime / history	version-controlled SSOT + 7-state machine + per-state edit rules	repository files (one has no default persistence model)
Human approval location	requirements layer only	many placements, or unspecified
External tracker	two adapters demonstrated + external-team read-only ownership model	built into one only; others absent or unconfirmed
Portability	2 agent CLIs (a fresh-repo port rehearsal passes with zero harness edits)	30+ / 40+ / 11 platforms — on the agent-CLI axis isopod is clearly behind
Onboarding friction	high (config → spec → anchor → cards → run)	mostly one command — isopod is clearly behind
Self-improvement	incident → rule → partial promotion to audit code	skill accumulation, or none

Where the Adjacent Tools Are Better (Explicitly)

One tool supports 30+ agents, is maintained by a company, and shipped 10 releases in two weeks. Another has low dependencies — no API keys, no MCP — and a design that pins exit codes and JSON shapes in a contract document; isopod has no equivalent document. The skill plugin family has dense approval points, so a first-time user never loses a sense of control. One autonomous SWE agent has tracker integration built in, solving as a product what isopod builds and maintains by hand. The orchestration framework offers durable execution, per-node retry policies, and interrupt/resume as first-class framework concepts; isopod hand-rolled the same things each time.

What isopod Deliberately Did Not Do

Low onboarding friction, IDE integration, broad CLI support, enforcing a spec document format (EARS or similar), and a general orchestration layer for arbitrary graphs. The pipeline is fixed.

What is the same: spec-first structure, phase separation, enforced TDD, artifacts in the repository — none of this is isopod's invention; it is common practice in this field.

6. Quantified Impact

6-1. Throughput (Measured)

Metric	Value
Runs executed	14 (over roughly two months)
Distinct stories executed by agents	73
Passed verification	70 (95.9%)
Terminated as failed	3
Agent passes	385
Total turns	31,116
Peak concurrency	13 passes (1.81× average)

Direct evidence of the frontier draining: the number of cards prepared per run declined monotonically — 101 → 100 → 99 → 80 → 80 → 80 → 78 → 74 → 70.

How to read the 95.9%. This pass rate must be read together with the waiver surface in §4-4. The gates do execute, but the agent can turn on N/A determinations for L3/L4 itself, so 95.9% does not mean "70 cards passed all four layers." It means "70 cards passed the set of gates that applied to each of them." The gate combination actually applied varies from story to story.

And these 70 cards predate both the §9-2 verification fixes and the exemption ledger. The fixes secure stories arriving from here on; they do not retroactively strengthen these 70, and there is no post-fix sample yet. This document does not blend the two points in time.

How to read concurrency of 13. Parallel execution is not the default. The worker count defaults to 1, and parallelism only turns on when the sweep sets an environment variable. Parallelism also happens between stories — the five personas within one story run sequentially. Isolation is at worktree level, not process or checkout level: N workers share a single target checkout, and only the commit window is serialized by a mutex.

What was filtered out to arrive at 73/70. A naive count reports verified: true 114 times. That includes (a) 44 re-run no-ops on stories already committed, (b) duplicate retry snapshots after failures, and (c) 8 cards a human implemented directly. Item (c) was excluded by exhaustively checking result files that record "runner not used" about themselves. This distinction is also the backbone of §8.

6-2. Cost and Time (Measured)

Item	Value
Main tally (385 passes)	$2,754
Retry snapshots (158 passes)	$1,257
Total spend	$4,012 (retries are 31.3%)
Per verified story	$39 ($57 including all retries)
Agent compute time	144.4 hours
Machine-live time	84.8 hours (1.70–1.81× parallel)

Cost by persona: implementation 34% · test authoring 24% · spec 18% · planning 18% · review 6%. Most of the verification budget goes into writing tests up front, not into review after the fact.

Context reuse: roughly 4.84 billion tokens were read from cache against roughly 163 million written to it — a reuse rate of about 96.7%. That is direct evidence the pipeline ran on accumulated context rather than rebuilding prompts each time. (Note that this cache effect is provided by the CLI; it is not a compression algorithm isopod implemented.)

Failure classification: of 63 retries, 43 (68%) were transient transport-layer errors, not logic failures. That reads two ways at once — without the retry machinery, that 68% would have been pushed onto a human, and simultaneously 31% of total spend went to operational stability rather than model quality.

6-3. Labor-Cost Conversion (Estimated — Assumptions Fully Disclosed)

What follows is an estimate. The only measured quantities are agent cost, time, and counts. Figures are shown in USD with the original KRW in parentheses.

Assumption	Value
A1 Human implementation effort per story	conservative 4h / mid 8h / optimistic 16h
A2 Fully-loaded hourly cost	$44 / $59 / $74 (₩60,000 / ₩80,000 / ₩100,000)
A3 Work a human still does per story	3h / 2h / 1h (spec, card review, PR review, merge, gates)
A4 Exchange rate	₩1,350/USD

Agent cost including retries is $57 per story (≈ ₩77,400).

Scenario	Hours replaced	Labor value	Agent cost	Net
Conservative	1h	$44	$57	−$13 (a loss)
Mid	6h	$356	$57	+$299
Optimistic	15h	$1,111	$57	+$1,054

Across 70 stories: −$0.9K to +$73.8K; the mid scenario is +$20.9K (≈ ₩28.2M).

The negative in the conservative scenario is not being erased. When story units are small and human residual work is large, this harness is economically a loss. It becomes favorable when (a) story units are large enough, (b) many cards run in parallel, and (c) traceability itself is assigned value.

Effects deliberately left unquantified: parallelism (peak 13 concurrent passes), traceability (200 cards, 438 dependencies, 1,018 subtasks, and 673 anchor references surviving under version control), and overnight operation. These are not converted into numbers, specifically to avoid inflating impact with an unfounded multiplier.

7. Incidents and Structural Fixes

Four cases were chosen so that no two share a failure class. All four were structural defects, not individual mistakes.

7-1. 21 Cards That Mistook "Merged" for "Works" — A Verification Gap

Symptom. The first time the integration demo was actually run, the major use cases all failed. Yet 21 cards on that path were already done.

Measurement. History search showed the defect was not a regression — it existed identically in the previous release. This path had never worked. The decisive measurement was "how many tests cross the boundary?": 8 tests that turn the parser's output into the internal shape, 6 tests that use the wire schema, and 0 that use both. Test each side separately and it stays green forever.

Easy misdiagnoses. "A recent merge caused a regression" (rejected by history search); "there weren't enough tests" (there were tests on both sides, and all were green).

Actual cause. The only gate approving published → done is merge-verify, and between merge and done there is no step that runs the merged code at all. Runtime truth arrives only through a human-opened observation gate — and that gate sat at the very end of the pipeline. So "the number of cards behind one gate" equals "the number of cards that can reach done on a foundation nobody has ever executed."

Structural fix. (1) An audit that counts cards behind an observation gate and warns past a threshold, with the threshold living in the config file rather than in code. (2) A design rule placing a thin vertical slice gate up front, so breakage surfaces at card 2 rather than card 23. (3) When opening an observation gate, contract-sweep freshness is enforced: if the covered paths changed since the last sweep, the gate opening is refused. There is exactly one bypass, and using it is recorded in the gate's evidence. (4) A single thin-vertical-slice smoke run was inserted between merge-verify and the state transition, so runtime truth enters the pipeline without waiting for a human to open a gate — though a failure does not block the transition. Blocking would directly reproduce §7-2, so the design stops at stamping done as evidence-less and surfacing it in the report. That choice — surface, not block — is what produces the residual limitation in §9-1.

Is it auto-detected — and the physical evidence. Yes. And the incident is still visible in the snapshot: this audit actually fires on the 200-card SSOT, where one gate holds 23 cards against a threshold of 8.

The core of this case: it was not a rule violation but the result of following the rules. Every merge-verify passed. The fix, too, went to a measurable threshold plus an automatic warning rather than to exhorting people to be careful.

7-2. A Pipeline That Stalled Waiting for a Human — Gate Placement

Symptom. The laptop battery died while unattended; on resuming, the runner had made no progress and was idle. Six drift approvals and two sign-offs were waiting for a response.

Measurement. Idle ratio per run (share of elapsed time with no events at all) reached 91% · 85% · 85% · 84% · 73% in some stretches. Others ran at 3% idle. The dominant variable in throughput was not model speed but human response time.

Easy misdiagnosis. "The battery caused it" — at the moment of discharge the runner was not computing but waiting. It was the waiting, not the power, that burned the battery.

Actual cause. Drift signals are structurally common in a parallel pipeline. A sibling story's commit inside the same service is enough to mark a dependency path "touched," because scope computation unions the target paths of dependency cards. In other words we had built a gate that summons a human on normal behavior.

Structural fix. Drift was demoted from an approval gate to an input annotation for the spec agent. Only structural errors (declared-but-absent, unmet dependency) abort; everything else routes automatically to a "rewrite while preserving existing exports" path. The old interactive flow survives only as an environment-variable opt-in. At the same time, the five downstream defenses expected to cover the lost signal were made explicit.

Is it auto-detected. No. This incident was absorbed as a design principle ("do not add new approval gates to the execution layer"), and its force is a code-review convention rather than a code gate.

The lesson is not "eliminate HITL" but "which layer you put HITL in determines throughput."

7-3. Layering Regexes Over Free Text — LLM Non-Determinism

Symptom. A post-processing step that decided whether a gate passed kept slipping on new phrasings each round. Widening the pattern produced yet another new phrasing next round.

Measurement. Refutation review surfaced 3–5 new missed channels every round. The whitelist kept growing but ultimately covered only half.

Easy misdiagnosis. "Widen the pattern further" — impossible in principle. The next sentence a model produces is a subset of all learned language, so no closed, complete set exists. Worse, a pattern miss was not a safe fallback but skipped logic — a silent false-green.

Actual cause. Taking non-deterministic free text as the input to deterministic post-processing at all.

Structural fix. Branching was moved onto schema-forced output (§4-7) and codified as a rule.

Is it auto-detected — and an honest refutation. There is no detection automation. And the investigation found the rule is not fully observed (§4-7, "where this must not be overstated"). Establishing a principle and having it upheld are separate things, and recording that gap in the document is itself the conclusion of this case.

7-4. A Card Whose Prose Was Fixed but Whose Graph Was Not — A Blind Spot No Validator Can See

Symptom. Computing the runnable set to carry a defect fix into the next execution, the card was not there.

Measurement. Of 71 published cards, 12 entered the frontier and 59 were blocked. Three were blocked solely by an observation-gate dependency, and in the extreme case a card's only dependency was a gate, so it could not enter any execution until a human opened it.

Easy misdiagnosis. "The consistency check passes, so the graph is fine" — that is exactly the trap. Every edge points at a card that exists; there are no cycles, nothing dangling, nothing orphaned. The graph is perfectly valid. It was simply encoding a plan that was no longer true.

Actual cause. Two layers of stale edges. (1) One decision superseded another and inverted a phased plan; the title, summary, and subtask prose was rewritten to the new plan, but the dependency array was left alone. (2) A gate's scope of application was narrowed, but cards outside the narrowed scope kept the gate dependency.

A structural asymmetry compounds this — code dependencies resolve within the same execution, but observation-gate dependencies cannot resolve in principle. The classifier that decides what an execution may satisfy skips observation gates, so in-run satisfaction is impossible and only a human can open them.

Structural fix. (1) A new rule — "when a decision is superseded, fix the dependency edges in the same commit" — which together with the existing retirement and new-decision rules covers all three axes: card creation, deletion, and replacement. (2) The verification method itself changed: since prose review cannot see this, compute the runnable set for real and check that the intended card appears. If it does not, the blocking reason names the cause.

Is it auto-detected — only partly. The retirement and new-decision axes have audit functions. But the stale-edge-after-supersession axis is still not auto-detected. "Is this edge still true?" is a semantic question, unreachable by schema validation or graph consistency in principle.

The defense did move from a procedural habit to a tool, however. Right after cards are published, the pipeline automatically checks whether the intended card actually enters the runnable set and reports the blocking reason (missing_deps) when it does not. A person's habit does not transfer to the next person; a tool does. The distinction remains that what got automated is "was the intent reflected in the graph" — not "was the intent correct."

7-5. The Common Structure Across the Four

#	Failure class	Easy misdiagnosis	Nature of the fix	Auto-detected
1	Verification gap	"tests are green, so it's fine"	measurable threshold + config value + post-merge smoke	Yes
2	Gate placement	"the battery caused it"	relocate gates by layer	No (design convention)
3	LLM non-determinism	"widen the pattern"	replace the input via schema forcing	No (partially unobserved)
4	Validator blind spot	"the check passes, so it's fine"	new procedure → tooling + partial audit promotion	Partial

Two of the four still were never promoted to automatic detection. That is how far this harness's self-improvement loop actually reaches. Case 3 in particular was found, after the principle was established, to be incompletely observed by the code — codifying an incident as a rule and having the rule upheld are separate things (§4-8).

8. What the Human Did Versus What the Harness Did

The first thing a reader of an agent-harness document suspects is "didn't a human actually do all of this?" The answer is not an overclaim but a precise boundary.

Span	Owner
Authoring natural-language specs	human (harness is read-only here)
Anchor extraction · card drafting · grouping	harness (LLM) + human review
Publishing to the tracker	human command (irreversible)
spec → plan → test → impl → review	harness (zero human gates) — 385 passes measured
PR review · merge	human
State transition · tracker sync	harness (human-invoked, conditional on merge-verify)
Post-merge smoke	harness (automatic) — story-level runtime truth. A failure stamps but does not block
Opening observation gates	human — the only point where integration-scenario runtime truth enters

What the Harness Did Not Do (Excluded by Measurement)

8 cards were implemented by a human directly. The result files record "runner not used," and they are excluded from the 73/70 tally in this document.
1 card had only its state-transition record hand-authored. The code itself landed through automated execution, so it was not excluded — and that distinction is recorded here too.

What This Boundary Means

What the harness automates is "reviewed card → verified commit." The front end that defines requirements and the back end that confirms operational truth are owned by humans. That is not a deficiency but a designed placement (§4-6, §7-2), and its justification came from an incident.

9. Limitations and Roadmap

9-1. `done` Still Does Not Fully Prove It Works

The most important limitation. A thin vertical-slice smoke run now sits between merge-verify and the state transition, so runtime truth enters the pipeline automatically — but a failing smoke does not block the published → done transition. All that remains is an evidence-less stamp on done and a line in the report.

This is a deliberate choice, not an omission. Blocking the transition would mean waiting for a human judgment on every failed smoke, which is a direct reproduction of §7-2 (a pipeline stalled on human response, 91% idle). So the accurate statement about the current design is "runtime truth arrives automatically and is surfaced automatically, but is never blocked automatically."

The residual axis is stated too — the smoke is one thin vertical slice, not full use-case coverage, and the truth about integration scenarios still enters when a human opens an observation gate. Even with the mitigations layered up (unverified-accumulation threshold warning, enforced contract sweeps, post-merge smoke), done cannot be equated with "it works."

9-2. The Hole in Verification Enforcement Is Closed — Timing and Residue Distinguished

An exhaustive code investigation for this portfolio surfaced four measured defects. They are published rather than hidden — and they were not left in place: all four were fixed, each with a boundary-crossing test that pins it. The real deliverable is that test, not the fix; fixing alone lets the same class recur at the next refactor.

Defect	Measured behavior	Fix + the test that pins it
Evidence completeness unchecked	The result-document validator did not pass the `rigorous` flag to the verification function, so a result with zero L1–L4 evidence exited 0	Pass the flag through, plus a test that spawns the CLI as a process and asserts exit 2 on a zero-evidence document
L2 mutation fail-open	A failure to spawn the test process was counted as "mutation killed" — the adjacent timeout branch guards against exactly that false accept (adjacent-branch asymmetry)	Route it to a harness error, excluded from the denominator with its own counter. Test: give a nonexistent command and assert the kill rate does not inflate
Stub guard dead code	The code tested `=== true` while the schema declares a string → the guard was unreachable	Conform the code to the schema. Test: a real row that passes schema validation actually trips the guard
Test/source separation bypass	The check keyed on a hardcoded path prefix, so edits to test files colocated in the source directory passed	Switch to the configured test glob. Test: editing a colocated test file is flagged as a violation. This row's closure was incomplete — see below

Three of the four shared one shape — both sides of a boundary are green while no test crosses the boundary. That is exactly the class §7-1 prescribed a remedy for, which means the harness was not applying to itself the discipline that rule demands of the target project (§4-8).

And "closed" was premature for the fourth row. Driving a greenfield spec through the pipeline afterwards (§9-3) found two defects the glob fix had revealed but not closed sitting behind it: the check compared against the roll base, making it impossible to pass under a TDD flow (paralysis, not enforcement), and git diff does not count untracked files, so a test the implementation authored was invisible (edits caught, authoring missed). Details and fixes are in the §4-2 overstatement note. The lesson here is not the defects but the detection method: all three came out of a full code audit, yet whether what that audit declared closed was actually closed could only be learned by running the pipeline end to end. A static audit answers "this code is wrong"; it does not answer "this fix works."

Two things remain, stated plainly.

Exemptions are counted, not blocked. L3/L4 N/A, L1 warning-downgrade, and L2 threshold reduction can still be turned on by the agent. What changed is that every exemption now passes through a single seam into a ledger with per-surface counts surfaced (§4-8). Why not block: the same reason as §9-1.
The fixes secure what comes next. The 70 cards in this document were verified before these fixes and before the instrumentation, so their verification strength cannot be retroactively upgraded in the telling.

9-3. Portability Was Demonstrated; Its Economics Were Not

Portability is the axis the entire payback argument rests on (§9-4), so it was treated as hypothesis testing rather than refactoring.

What was achieved: the card schema's ID-prefix regex and team enum are now synthesized from configuration at load time, so a freshly initialized project no longer produces schema-invalid cards (not an invention — the result schema already used the pattern of injecting enums from code into the schema to make drift structurally impossible; this filled in a place where that pattern had not been applied). The single-uppercase-letter constraint on card ID prefixes is lifted. A second tracker adapter is verified with real round trips. And as the decisive evidence — one card ran end to end in an empty repository, from initialization through anchoring and card creation to execution, with zero harness code changes.

The sample was later widened from one card to the whole chain. On the second project (a link shortener), the target repository was reset to its seed commit and every derived artifact deleted, leaving nothing but a single natural-language spec document, and the full chain was run: 19 anchors (agreed after one repair round) → 3 cards + 1 gate (all 19 anchors assigned to exactly one card each) → all 3 code stories verified → PR. The observation gate did not open itself; it waited for a human, as designed. Rather than trust the harness's own gates, the service was actually booted and each sentence of the spec checked over HTTP — 201 response shape, 6-character base62, 302 plus Location, idempotent creation, code uniqueness, 404 on an unknown code, and the 30-day boundary (29 days → 302, exactly 30 → 410, 31 → 410). All 13 claims held.

But "zero harness changes" no longer holds at this scale. Getting those three cards through took three roll attempts and four harness defect fixes: anchor adoption choosing the wrong prefix (§4-1), the diagnostic panel never being shown its own output schema, the test/source separation gate being unpassable under TDD and blind to untracked files (§4-2, §9-2), and a regression that fix introduced on a CRLF checkout. So the precise claim is this — portability with zero edits holds at one-card scale, and does not yet hold across the full path from natural-language spec to PR. In fairness, three of the four were not defects of porting but latent defects of the original project that first fired under the new layout (colocated tests, zero dependencies), and all four fixes landed in the harness rather than the target.

What was not achieved: this proves mechanical portability, not economics. The second project is a practice example, not a real one, at a scale of three cards. Whether a second real project yields savings remains unmeasured. Agent CLI support is still two (against 30+ / 40+ / 11 platforms for adjacent tools), and that is a deliberate non-goal — proving the adapter is genuinely swappable with two was judged more valuable than supporting N.

9-4. Two Questions This Document Cannot Answer (The Sharpest Objections)

Two questions this document cannot honestly answer are stated first, because dodging them would undermine everything else.

(1) "What did building the harness cost?"

It cannot be measured. The repository history was re-initialized partway through, so commit counts cannot be converted into effort, and development time was not recorded separately. What is clear is that building a 142K-LOC harness likely cost more than the estimated savings recovered on this single project (mid scenario ≈ $21K). The payback argument for this harness rests not on single-project savings but on reuse — porting to other projects. Per §9-3 that reuse is now mechanically demonstrated but still economically unverified — "a natural-language spec runs through to a PR on a second project" and "a second real project yields savings" are different propositions, and even that run-through required fixing four harness defects. So the claim "this harness paid for itself" still does not hold within the scope of this document.

(2) "Was the result the harness or the model?"

They cannot be separated. No ablation was run — there is no result for handing the same cards to the same model without the harness. So "95.9% is the harness's achievement" cannot be claimed.

What can be said is narrow: which spans could not exist without the harness. Computing dependency order between cards, automatic restart until the frontier empties, four-layer gates adjudicated by deterministic programs, the rule that state transitions require passing merge-verify, and the enum classification of failures with the retry budgets keyed to it — these were provided by the pipeline independent of model capability. Conversely, claims like "the code quality was good" or "the design judgment was sound" are inseparable from the model's contribution.

9-5. Sample and Evaluation Limits

14 runs / 73 stories / one real project (the second is a 3-card practice example and does not widen the sample — §9-3). Never placed on a public benchmark, so external comparison is impossible. There is no formal eval harness, golden dataset, or trajectory evaluation — the review agent is a judge, not an evaluation system.

9-6. Roadmap (In Priority Order)

The previous roadmap's five items (wiring up the evidence-completeness check · fixing the L2 mutation fail-open · removing project coupling from the schema · partially automating observation gates · demonstrating a second tracker adapter) are complete, joined by the exemption ledger and stale-edge tooling. What remains:

Let the exemption distribution pick the next axis. Once ledger data accumulates, it decides which defense line is nominal. Exemptions concentrated on one gate → the verification-depth axis; uniformly low across surfaces → the portability axis; concentrated in one story type → retarget at the spec-authoring stage. The branch rule was fixed before the data arrived — setting the rule after seeing the data would be post-hoc rationalization.
Widen smoke coverage — currently one thin vertical slice (§9-1). How far to widen depends on (1)'s data.
Port to a second real project — the economics argument hangs here (§9-3, §9-4). A full lap on an actual project, not a one-card rehearsal.
Semantic adjudication of stale edges — the principled residue of §7-4. There is still no way to auto-decide "is this edge still true."
Build a regression sample — there is no golden card set to catch regressions in harness changes themselves. Once (1)'s instrumentation accumulates, that data seeds a minimum sample.

10. Appendix

10-1. Technology Stack

Area	Composition
Runtime	Node.js (minimal dependencies, mostly standard library)
Data	one card = one YAML file; JSON Schema + ajv validation
Verification	TypeScript compiler API (static) · AST mutation + test runner · JSON deep-diff · process spawn
Isolation	git worktree per story, mutex-serialized commit window, N→1 commit squash
Observability	per-pass NDJSON event tee + per-backend normalizers
Abstraction	agent backend adapters (2) · tracker providers (2)
Static analysis	ESLint + sonarjs + jscpd, churn-weighted prioritization

Code size: 633 tracked files / 142K LOC. Source .js 254 files (68.0K LOC), tests 284 files (63.9K LOC), test:source LOC ratio 0.94.

10-2. Reproducing the Numbers

The agent-execution tally is regenerated by a single script, which handles three traps at the code level.

Trap	Naive result	Handling	Corrected
Re-run no-ops	`verified:true` = 114	split on skip-reason presence + dedupe by ID	85 distinct
Duplicate retry snapshots	543 passes / $4,012	separate from the main tally by path	385 passes / $2,754
Double-counted parallelism	reading 144.4h as elapsed time	count distinct active passes per 1-minute bucket	84.8h live, 1.81 average, 13 peak

Adding the exclusion of 8 human-implemented cards (§6-1) yields the final 73/70.

The concurrency metric was discarded twice before the third was adopted. (1) compute time / elapsed time mixed idle with parallelism and produced values below 1. (2) The union of each pass file's [first event, last event] interval inflated to 10× compute time, because retries append to the same file. (3) Adopted — fold events into 1-minute buckets and count the number of distinct passes that emitted an event in each bucket. It rests on no assumptions and agrees independently with 144.4/84.8 ≈ 1.70.

10-3. Investigation Method

The evidence for this document came from four channels: harness code (582 files at the time of the audit; 633 now), operating rule documents (29 rules), execution artifacts (3,081 files across 14 runs), and the SSOT snapshot (200 cards). A fifth was added afterwards — the greenfield run to completion on a second project (§9-3) — and it reopened one of the things the static audit had declared closed (§9-2).

Technical claims were verified by exhaustive code investigation, with the investigation brief specifying "no unsupported claims, file::symbol citations required, actively search for counter-evidence." As a result at least five draft claims in this document were refuted and corrected — that done-card immutability is a convention rather than code enforcement; that the evidence completeness check is not wired up; that "five personas in parallel" is actually sequential within a story; that the diagnostic panel has no majority vote; and that config-driven portability breaks at the schema layer.

Rather than deleting the refuted claims and publishing only what survived, the refutations were kept in the body. That is what the "where this must not be overstated" notes are.

And what could be fixed was fixed. Of those five, the unwired evidence-completeness check and the schema-layer coupling were corrected per §9-2 and §9-3; the other three were corrections of fact, so the sentences themselves were amended. To keep the investigation and the correction from blurring into one moment, every fixed item records both what was found and what closed it — deleting the finding would delete the methodology that made it findable.