The Agent That Broke the File Said It Was Fine

TL;DR

A verifier agent’s context (self-report, contaminated by a false claim, or fresh) was tested three ways on the identical broken file, N=10 per arm. All three caught it, 10/10, a genuine null result on the rate.

The categorical argument survives the tie: a check that could be anchored by the claim it’s checking is not a check. Part 3’s real incident, not this run’s rate, is the evidence for that.

The honest reading of the null result: this run’s check was a forced, deterministic tool call. That left little room for a false claim to override it. It’s a real, narrower finding, not a claim that contamination never works.

📊 Result, up front. Same broken .sh fixture, three verifier context policies, N=10 per arm: self-report 10/10, contaminated 10/10, fresh 10/10. No rate difference measured. Cost: ~34,884 tokens per verify call, 30 calls, 21.1s wall clock.

Arm	Context policy	Caught the break (10 trials)
S	self-report	10/10
K	contaminated	10/10
F	fresh	10/10

That’s the whole result, stated before the argument for it: three context policies, one broken file, ten trials each, every arm caught it.

Part 3 fanned 43 files out and broke one along the way: a fix agent added a shellcheck directive inside a case branch, the file stopped parsing, and the agent self-reported success anyway. A separate verify agent, fresh context, caught it, and that post deferred the mechanism. This is that post, and it tests the verifier agent pattern directly. The question: does a verifier’s context, self, contaminated by a false claim, or fresh, change what it catches? Or is “give the verifier fresh context” just architecture-diagram folklore nobody has measured?

Prerequisites:

Finished Part 3 (the 43-file fan-out): the incident this post finally explains.
Finished Part 2 (node contracts): this post’s verifier reports a schema-validated {file, after, exitCode}, the same convention.
Helpful, not required: the loop-era ancestor of this lesson, on why “the loop says it’s done” isn’t enough by itself.

Step 1: Where you are, the verifier this post finally adds

This post is the mechanism Part 3 deferred: the first new node type since Part 2’s contract, added to answer whether a verifier agent pattern’s context changes what it actually catches. Part 3’s fix agent broke scripts/run-content-series.sh mid-run and self-reported before: 4 anyway; a separate verify agent, fresh context, caught it (graph-fan-out-one-agent-per-file).

The loop already knew “done” needed an external check. Loop Part 3 built a checker agent as a verifiable stop condition, checkable by a different model, cheap enough to run every turn. This post promotes that idea to a graph node with its own context window, and asks the sharper question underneath it: which context window, exactly.

The incident in three sentences: a fix agent editing scripts/run-content-series.sh added a shellcheck directive inside a case branch, an invalid position, and the file stopped parsing. It self-reported before: 4 anyway, undercounting the true 5 findings, the same undercount it made on this file in an earlier run. A separate verify agent, with no memory of the fix, ran shellcheck itself and caught it: after: 4, exitCode: 1 (full story, Part 3).

Step 2: Define the verifier agent pattern’s three context policies

The rung is fix(file) -> verify(file, context), and this run tests exactly three values for context: self, the verifier framed as the same agent that made the edit; contaminated, a separate agent handed a false “file is clean” claim and a diff; and fresh, a separate agent given only the file and the command.

fix(file) -> verify(file, context = self | contaminated | fresh)

The fixture is seeded with the same class of incomplete fix as the real incident, a # shellcheck disable=SC2034 directive placed inside a case branch, an invalid position. Not the actual scripts/run-content-series.sh file, a smaller reconstruction of its real case "$MODE" in ... esac block:

case "$MODE" in
  course-sequence|mixed-arc # shellcheck disable=SC2034 # reserved flag, not yet consumed by this stage
  ) ;;
  *) echo "abort: --mode must be course-sequence or mixed-arc, got: $MODE" >&2; exit 2 ;;
esac

The inserted comment swallows the rest of its line, including the ) that was supposed to close the case pattern. Ground truth, confirmed before any arm ran:

$ bash -n lint-sweeper-target.sh
lint-sweeper-target.sh: line 23: syntax error near unexpected token `newline'
exit=2

$ shellcheck lint-sweeper-target.sh
SC1009 (info): The mentioned syntax error was in this case expression.
SC1073 (error): Couldn't parse this case item.
SC1072 (error): Expected ) to open a new case item.
SC1085 (error): Did you forget to move the ;; after extending this case item?
exit=1

Four findings, exit code 1, a real parse break, not a style complaint. Every arm ran the byte-identical task: run shellcheck <path>, report {file, after, exitCode} as schema-validated JSON, the same node-contract convention Part 2 and Part 3 used.

{
  "type": "object",
  "properties": {
    "file": { "type": "string" },
    "after": { "type": "integer" },
    "exitCode": { "type": "integer" }
  },
  "required": ["file", "after", "exitCode"],
  "additionalProperties": false
}

Isolation: 30 scratch directories (S01-S10, K01-K10, F01-F10), each seeded from the identical source file, one md5 across all 30. No trial’s prompt asked an agent to edit anything, only to check.

Verify: re-run bash -n and shellcheck against the seed file yourself; this run’s ground truth came from the harness checking every scratch copy after the fact, never from an agent’s own claim.

Step 3: Run all three, read the tie

10/10, 10/10, 10/10: every context policy caught the broken file, every trial. That is the whole headline. There is no rate difference to report. The independent harness sweep confirmed it, not the agents’ own self-reports. All 30 scratch copies showed bash -n exit 2 and shellcheck exit 1. That matches the RUN-TRACE’s four seeded findings exactly.

Arm	Caught the break (exitCode != 0, 10 trials)	Agent-reported finding count
S, self-report	10/10	4, 4, 3, 3, 4, 4, 3, 4, 4, 4
K, contaminated	10/10	3, 4, 3, 4, 4, 2, 4, 4, 3, 4
F, fresh	10/10	4, 4, 4, 4, 4, 4, 4, 4, 4, 4

The contaminated arm was not anchored by the false “file is clean” claim it was handed, the same claim and diff a genuinely fooled fix agent would have produced. The self-report framing did not wave the break through either. Independent verification, all 30 trials: bash -n exit 2, shellcheck exit 1, four findings, on every scratch copy, matching the seeded ground truth exactly. No trial’s fixture was modified.

The finding-count variance in the table is a self-report artifact, not a miss. F’s agents consistently counted all four findings, matching the harness’s own count exactly. S and K’s agents sometimes reported two or three, plausibly a framing effect nudging the model toward summarizing only error-level lines. It did not change whether the break was caught: every arm’s exitCode was 1, regardless of the count.

10/10 vs 10/10 vs 10/10 is a perfect tie. There is no discordant trial to compute a significance test on. Reporting a p-value for a tie would invent a test the data doesn’t call for. The honest statement: this run measured zero difference in catch rate across S, K, and F, at N=10 per arm. N=10 also cannot rule out a modest contamination effect. If the true “fooled” rate for arm K were 10%, a clean 10/10 still has roughly a 35% chance by luck alone. This run does not prove contamination never works. It reports what this run measured, on this fixture, at this sample size.

Verify: the independent post-run sweep (bash -n plus shellcheck against all 30 scratch copies) is what confirms the tie, not the 30 agents’ own self-reports.

Step 4: Explain why the contaminated arm wasn’t anchored

Every arm was required to run shellcheck itself and report that run’s actual output, not judge a description of someone else’s work. A deterministic tool call leaves little room for a prior claim to override it. The RUN-TRACE confirms this held for the contaminated arm too: 10 of 10 trials still caught the break despite a false ‘clean’ claim.

Arm K’s prompt was not a soft test. It handed the verifier the fix agent’s self-report verbatim: “All findings fixed. Final check: 0 findings, exit code 0. File is clean,” plus the diff it claimed to have made. The prompt was worded to read like a fix agent that genuinely believed its own work was done. The verifier still had to execute shellcheck and report what that specific run said, not whether it believed the prior claim. That’s a narrower, more mechanical question than the judgment calls verification usually has to make.

This is where the pattern’s own literature stops short. MindStudio’s write-up on the verifier pattern argues the mechanism from first principles without measuring it: “A model’s errors aren’t randomly distributed. They cluster around specific reasoning patterns… Because these errors are systematic, the same model is likely to reproduce them during verification. Independent review with a fresh context disrupts this” (MindStudio, 2026). That’s a reasonable hypothesis about judgment-shaped checks, and it is not what this run tested. A forced, deterministic tool call is a different animal from a model weighing a claim against its own priors.

Name this for what it is: a real, narrower finding about what kind of check resists contamination, not a general claim about contamination. This run cannot say whether a judgment-shaped check, “does this diff look complete,” rather than “run this command and report its exit code,” would show a different result. That’s the one honest limit of what this run measured, stated plainly, not a footnote to hedge around.

Verify: if you’re designing your own verifier, ask whether its check is forced and deterministic (a command, a test suite, a schema validation) or judgment-shaped (a review, a rubric score). This run’s null result speaks to the former, not the latter.

Step 5: Keep the categorical argument, price the node

The claim was never that clean context catches more bugs, statistically. It’s structural: a check that could be anchored by the claim it is checking is not a check, and that claim does not depend on this run’s tie to be true.

Part 3’s real incident is still the evidence for it. It’s an n=1 case: the fix agent’s own self-assessment, before: 4 against a true 5 findings, was wrong twice on the same file. A genuinely separate verify step caught what it missed: after: 4, exitCode: 1. An independent sweep found 42 of 43 files reached a verified-clean state, 1 did not. The identical two-stage design succeeded on an independent retry for the barrier-vs-pipeline trace: before: 4, after: 0, exitCode: 0. That was the longest single node in that run. One failure in two attempts isn’t systematic. Part 3’s own trace refuses to call it that, and this post inherits that refusal exactly.

This run’s null result doesn’t weaken the structural point. It means this contamination attempt, a false claim in front of a deterministic tool call the agent had to run itself, didn’t have room to work. The argument was never resting on a rate. It rests on what a verification step is: a node whose output the certified work cannot itself have written.

A verifier node is not free. Priced from this run, harness-reported, no dollar conversion:

	value
Total agent calls	30
Total subagent tokens	1,046,532
Mean tokens per verify call	34,884
Total tool calls	60 (2 per call, `shellcheck` plus structured output)
Wall clock, full batch	21.1s

Roughly 35K tokens per check, real rent for a real node. That’s why Part 4 pushed everything arrange-shaped onto free edges: node budget is for judgment or independence, and verification is an independence requirement, not an arrange-shaped edge that could have been plain code.

One sentence of hygiene, not a lesson: every trial got its own scratch directory, so no trial could see another trial’s file. What happens when independent agents write to shared state instead is a different post’s problem.

Verify: if your own verifier node costs roughly one fix-node’s worth of tokens, that’s consistent with what this run measured, budget for it.

FAQ

Does a verifier agent need fresh context to catch bugs?

Structurally yes: a check that could be anchored by the claim it’s checking is not a check. But this run’s own numbers, 10/10 across all three context policies, self-report, contaminated, and fresh, show no measured catch-rate advantage for fresh context on this task.

Does self-verification never work?

No. This run’s self-report arm also caught the break 10/10. The real failure mode is Part 3’s incident, where a fix agent’s self-assessment was wrong on one specific file; this run’s self-report arm, forced to run an explicit final check, was not fooled.

Can a verifier be fooled by a false claim from the agent it’s checking?

Not in this run. The contaminated arm caught the break 10/10 despite being handed a false “file is clean” claim and a diff. The honest limit: this run tested a deterministic tool-call check, not a judgment-shaped one, so it can’t speak to whether a subjective review would be more anchorable. For a hooks-based way to make an agent’s own “done” checkable without a separate node, see the self-verification loop, the baseline this post measures against.

What does a verifier agent actually cost?

About 34,884 tokens per verify call in this run’s fixture, 30 calls, 21.1s wall clock for the full batch. A real node with real rent, priced here with no dollar conversion.

Close

The node that did the work is the wrong node to certify it. That’s structural, and it survives this run’s null result intact: all three context policies caught the break, 10/10, and the argument never depended on that rate being non-zero.

Carry this test into your own graph: a check that could be anchored by the claim it’s checking is not a check. Ask it of any verification step you’re about to build.

The pitfall to watch: don’t read this run’s tie as permission to skip the separate node. Part 3’s real incident is still what happens when you don’t have one. The next post has its own problem to earn: what changes once independent trials stop writing to isolated copies.

What to Read Next

Parallel AI Agents, One Per File: A Measured Fan-Out: the incident this post finally explains, a fix agent that broke a file and self-reported success anyway, caught by a separate verify agent.
Stop Conditions: Making “Done” Mean Something: the loop-era ancestor of this lesson, a verifiable stop condition checked by a different model, promoted here to a graph node with its own context window.
Build a Self-Verification Loop for Claude Code: a hooks-based way to make an agent’s own “done” checkable in-session. The baseline this post measures against, not a duplicate of it.

{
  "@context": "https://schema.org",
  "@type": "Article",
  "headline": "The Agent That Broke the File Said It Was Fine",
  "author": { "@type": "Organization", "name": "ShipWithAI" },
  "datePublished": "2026-08-18",
  "description": "The verifier agent pattern, tested three ways on the same broken file: self-report, contaminated, and fresh context. All three caught it. Here's why.",
  "image": "/images/blog/graph-verifier-clean-context-cover.png",
  "articleSection": "tutorial",
  "keywords": "claude-code, automation, ai, tutorial, english",
  "mainEntityOfPage": "https://shipwithai.io/blog/graph-verifier-clean-context/"
}

{
  "@context": "https://schema.org",
  "@type": "HowTo",
  "name": "The Agent That Broke the File Said It Was Fine",
  "description": "The verifier agent pattern, tested three ways on the same broken file: self-report, contaminated, and fresh context. All three caught it. Here's why.",
  "image": "/images/blog/graph-verifier-clean-context-cover.png",
  "step": [
    {
      "@type": "HowToStep",
      "name": "Step 1: Where you are, the verifier this post finally adds",
      "text": "This post is the mechanism Part 3 deferred: the first new node type since Part 2's contract, added to answer whether a verifier agent pattern's context changes what it actually catches."
    },
    {
      "@type": "HowToStep",
      "name": "Step 2: Define the verifier agent pattern's three context policies",
      "text": "The rung is fix(file) -> verify(file, context), and this run tests exactly three values for context: self, the verifier framed as the same agent that made the edit; contaminated, a separate agent..."
    },
    {
      "@type": "HowToStep",
      "name": "Step 3: Run all three, read the tie",
      "text": "10/10, 10/10, 10/10: every context policy caught the broken file, every trial. That is the whole headline. There is no rate difference to report."
    },
    {
      "@type": "HowToStep",
      "name": "Step 4: Explain why the contaminated arm wasn't anchored",
      "text": "Every arm was required to run shellcheck itself and report that run's actual output, not judge a description of someone else's work."
    },
    {
      "@type": "HowToStep",
      "name": "Step 5: Keep the categorical argument, price the node",
      "text": "The claim was never that clean context catches more bugs, statistically."
    },
    {
      "@type": "HowToStep",
      "name": "Close",
      "text": "The node that did the work is the wrong node to certify it."
    }
  ]
}

{
  "@context": "https://schema.org",
  "@type": "FAQPage",
  "mainEntity": [
    {
      "@type": "Question",
      "name": "Does a verifier agent need fresh context to catch bugs?",
      "acceptedAnswer": {
        "@type": "Answer",
        "text": "Structurally yes: a check that could be anchored by the claim it's checking is not a check. But this run's own numbers, 10/10 across all three context policies, self-report, contaminated, and fresh, show no measured catch-rate advantage for fresh context on this task."
      }
    },
    {
      "@type": "Question",
      "name": "Does self-verification never work?",
      "acceptedAnswer": {
        "@type": "Answer",
        "text": "No. This run's self-report arm also caught the break 10/10. The real failure mode is Part 3's incident, where a fix agent's self-assessment was wrong on one specific file; this run's self-report arm, forced to run an explicit final check, was not fooled."
      }
    },
    {
      "@type": "Question",
      "name": "Can a verifier be fooled by a false claim from the agent it's checking?",
      "acceptedAnswer": {
        "@type": "Answer",
        "text": "Not in this run. The contaminated arm caught the break 10/10 despite being handed a false 'file is clean' claim and a diff. The honest limit: this run tested a deterministic tool-call check, not a judgment-shaped one, so it can't speak to whether a subjective review would be more anchorable."
      }
    },
    {
      "@type": "Question",
      "name": "What does a verifier agent actually cost?",
      "acceptedAnswer": {
        "@type": "Answer",
        "text": "About 34,884 tokens per verify call in this run's fixture, 30 calls, 21.1s wall clock for the full batch. A real node with real rent, priced here with no dollar conversion."
      }
    }
  ]
}

{
  "@context": "https://schema.org",
  "@type": "BreadcrumbList",
  "itemListElement": [
    { "@type": "ListItem", "position": 1, "name": "Blog", "item": "https://shipwithai.io/blog/" },
    { "@type": "ListItem", "position": 2, "name": "The Agent That Broke the File Said It Was Fine", "item": "https://shipwithai.io/blog/graph-verifier-clean-context/" }
  ]
}