Agent Loop Reliability: The Math That Demands Verification

TL;DR

  • Per-step reliability compounds against you: a loop that is 95% reliable per step succeeds about 60% of the time over 10 steps, 36% over 20, and is below a coin flip by step 14.
  • Verification is load-bearing. A verified checkpoint resets the chain: the loop only has to survive the segment between checkpoints, and a detected failure becomes a bounded retry instead of silent compounding.
  • Test feedback also raises per-step reliability itself, which is why coding agents worked first, and why deterministic criteria plus a separate evaluator (the official /goal design) is architecture rather than a best practice.

Multiply 0.95 by itself twenty times. You get 0.358. That one line is the case for everything this series has asked you to build: an agent that succeeds 19 times out of 20 on any single step completes a 20-step run about one time in three. Agent loop reliability is a property of the chain, and chains are longer than they feel. ByteByteGo’s recent essay The Agent Loop deserves credit for making this framing stick: a loop turns a per-turn model into a multi-step system, and the system pays compound interest on every per-step flaw.

So the thesis, stated plainly: verification is a mechanism, not a nicety. A verified checkpoint is the structural lever that changes the equation an unattended loop lives under, and the practices the Loop Engineering series teaches are consequences of that one equation. Part 3 showed you how to write verifiable stop conditions. Part 9 showed you where loops fail and made you cap them. Neither derives why those disciplines are load-bearing rather than hygienic. This appendix exists to supply the derivation.

The stakes are concrete and probably familiar. Your loop nails eight steps and dies at step twelve. You blame the model, tighten the prompt, and retry, and the next run dies at step nine. The math below says the model was never the main problem; the run length was. If you have never multiplied 0.95 by itself twenty times, do it once. Everything the series taught, stop conditions, maker and checker, caps, snaps into place as consequences of one number instead of a list of best practices.

The Compounding Math Behind Agent Loop Reliability

Reliability multiplies across steps instead of averaging. If each step succeeds with probability p, a run of n independent steps succeeds with probability p^n, and p^n collapses earlier than intuition expects. Per the compounding framing in ByteByteGo’s The Agent Loop, 95% per step yields 59.9% over 10 steps and 35.8% over 20.

Be honest about what a step is, because that is where the 95% hides. A step is any unit of work that can silently go wrong: a tool call that returns stale output and gets trusted anyway, a file edit that lands in the wrong function, a judgment call that picks the plausible answer over the correct one. Each rolls the dice. For an agent doing real work, 95% per roll is a generous grant, and the argument can afford to be generous because it wins anyway.

One table is the entire apparatus of this essay. No second calculation is coming; every section that follows reuses these eight numbers.

Steps (n)p = 0.95 per stepp = 0.99 per step
100.5990.904
140.4880.869
200.3580.818
500.0770.605

Walk the left column honestly. Ten steps in, you hold 59.9%. Step 13 puts you at 51.3%, and step 14 at 48.8%. A loop that is 95% reliable per step is below a coin flip by step 14. At twenty steps you hold 35.8%, and at fifty, 7.7%, lottery territory for the kind of run people now leave unattended overnight.

Two caveats before these numbers do more work. First, they are consequences of a stated model and nothing else. Nobody measured a 35.8% success rate; this essay claims no telemetry, and every value above is arithmetic you can check on a napkin. Second, the model assumes steps fail independently, and real steps behave worse, because one bad step poisons the steps after it. The steelman section takes that objection seriously; for now, note only that independence is the optimistic assumption.

Verification Shortens the Chain a Loop Must Survive

Verification’s real job is resetting the chain mid-run, which matters more than grading the final output. Add a trusted checkpoint every k steps and the run never has to survive more than k unwatched steps. The governing math shifts from p^n to p^k with retries, and per the table above, that shift is what a checkpoint buys.

This is why the series’ canonical definition refuses to treat verification as an accessory:

A loop is a recursive goal: you define a purpose and a verifiable stop condition, and the system iterates agents against it until the condition holds — without you prompting each turn.

Read against the table, the wording earns itself. A loop without a verifiable stop condition is the left column: an unwatched chain decaying geometrically, reporting success at the end because nothing along the way ever forced it to prove anything.

Reuse the numbers. Surviving 20 unwatched steps runs at roughly 36%. Break the same run into four verified segments of five steps each and every segment survives at about 77%, the same arithmetic applied to a chain a quarter the length. The decisive difference is the failure shape. In the unwatched run, an error at step six travels silently through fourteen more steps, and everything downstream is built on a wrong foundation. In the checkpointed run, the segment boundary catches it, and the cost is a bounded retry of at most five steps.

Both failure shapes exist in the wild. Silent compounding is what Anthropic documents in Effective harnesses for long-running agents: premature victory declarations, features marked done that were never wired up, the agent asserting success because asserting is cheaper than achieving. Bounded retry is what you buy with a checkpoint. Failures still arrive at the same per-step rate, but each one is detected, localized, and priced.

The HOW is deliberately out of scope here. Verifiable stop conditions covers the discipline of writing checks that actually verify, from deterministic criteria up. This essay’s job is the WHY underneath it: whatever your checkpoint discipline costs, the left column of the table is what it is saving you from.

Test Feedback Raises Per-Step Reliability

Verification attacks the equation from both sides. Chain-shortening reduces n; verifier feedback raises p itself, because a step that sees a failing test corrects course before its error ships downstream. That second effect is ByteByteGo’s other observation in The Agent Loop: test feedback raises per-step reliability, which is why coding agents worked first.

Coding got lucky as a domain. It shipped with cheap, deterministic, machine-checkable verifiers preinstalled: tests, compilers, linters, type checkers. An agent editing code gets a truth signal in seconds, per step, at near-zero marginal cost, and the self-verification loop is that advantage exploited deliberately inside a single session. Domains without built-in verifiers, prose, design, strategy, are the hard tail for exactly this reason: their p stays wherever it started, and chain-shortening is the only lever left.

The table prices what a verifier is worth. Read across the rows: 0.99^50 is 0.605, essentially the same odds as 0.95^10 at 0.599. Raising per-step reliability from 95% to 99% buys roughly five times the chain length at the same odds. That is the exchange rate, and it is what a good verifier does to your p every time a failing check turns a shipped error into a corrected one. In practice, the two effects compound: feedback raises p within a segment while checkpoints keep segments short.

And still the 0.99 column decays. By step 50 the better model is back where the worse one stood at step 10. Better steps postpone the cliff without removing it, which is the cleanest one-line argument that model upgrades and verification are complements.

The Steelman: Real Steps Are Not Independent

The p^n model is wrong in three honest ways, and every one of them strengthens the conclusion. Any engineer who has run real loops will raise all three unprompted, so they deserve full concessions before any rebuttal.

Steps are not independent coin flips. Conceded. One bad step poisons the context for every step after it: a wrong assumption gets written into a file and becomes load-bearing, a hallucinated API shape becomes the substrate of the next three edits. But correlated errors make unverified chains decay faster than p^n predicts, because failures cluster and cascade instead of arriving alone. The independence model is the optimistic bound, so the argument survives its own simplification. The simplification was flattering the opposition.

Verifiers are imperfect. Conceded. Real checks produce false passes and false failures, and a maker grading its own homework is the opening problem of the stop-condition discipline. The lineage of the fix predates the loops wave: Anthropic’s Building effective agents described the evaluator-optimizer pattern, a separate evaluator grading a generator’s output, back in 2024, and the official @ClaudeDevs loops post records the same design choice, preferring deterministic criteria where they exist and delegating the rest to a separate evaluator model. None of that makes a verifier perfect. An imperfect checkpoint still bounds how far an error can travel; the alternative bounds nothing.

A better model raises p. Conceded, and the table already priced the purchase: 0.99^50 is 0.605, so a 99% per-step loop holds at step 50 the odds a 95% loop holds at step 10. Chain length is worth buying. But the decay law never repeals, and model quality composes with verification instead of competing against it. Raise p with a better model, shorten n with checkpoints, and both terms of the equation finally work for you.

From Math to Mechanism: Stop Conditions and Caps

The series already shipped the answer to this equation, in two halves. The stop-condition discipline installs the checkpoint: deterministic criteria where possible, a separate evaluator model where not, which is the design the official /goal command uses to grade its condition. The caps make the retry affordable: bounded iterations, bounded cost, bounded time.

Seen from the table, the two halves stop looking like independent tips. A stop condition chooses where the chain resets; a cap bounds what a reset can cost. A retry loop without a cap is the unbounded-iteration failure mode wearing a lab coat, which is why the failure modes and the three caps belong in the same argument as the stop-condition discipline. The official loops post names four loop types and records the same grading order the series teaches: deterministic criteria first, an evaluator model for the rest. The era where the series started is one of loops that run while you sleep; the arithmetic above decides whether they finish.

Close by inverting the opening number. 0.95^20 is 0.358, and no prompt, system message, or model upgrade changes the shape of that curve; every mechanism in this series exists to move one of its two variables. Feedback raises p. Checkpoints shorten n. Caps price the retries that checkpoints make possible.

The honest limit: p^n is a cartoon, nobody has measured your loop’s true p, and the cells of the table are illustrations rather than predictions. Trust the shape of the curve, distrust any individual cell. What to do differently is specific: stop treating the verifier as the final stage of the pipeline and start treating it as the thing that decides how long your chain actually is. The loop you can trust is the loop whose chain never gets long enough to lose.

FAQ

Why do agent loops fail more often as they run longer?

Per-step reliability compounds: a loop that is 95% reliable per step succeeds about 60% of the time over 10 steps and 36% over 20, because reliability multiplies across steps rather than averaging. Longer chains fail more often even when no single step got worse.

How does verification improve agent loop reliability?

Two ways. A verified checkpoint shortens the chain, because the loop only has to survive the steps between checkpoints and a caught failure becomes a bounded retry. Verifier feedback also raises per-step reliability itself, by letting a step correct course before its error ships downstream.

How many steps can an agent loop run reliably?

Unverified, a 95% per step loop drops below a coin flip by about step 14. With verified checkpoints the effective chain length is the checkpoint interval rather than the run length, which is why verified loops can run far longer than the raw math suggests.