Agent Workflow Observability: The Node That Died Quietly
Agent workflow observability, concretely: a fan-in guard that counts nodes returned vs expected so one dead node can't pass as a finished report.
tutorialAgent Workflow Observability: The Node That Died Quietly
TL;DR
In a chain, a failure is loud: C dies, D never runs, the run halts where you can see it. In a graph, one dead node among forty lands in a report that looks complete.
The fix is a fan-in guard you own, written into your reduce step. No dashboard you buy will surface this failure. Count results returned against nodes dispatched, and fail loud, naming the missing node, before you reduce.
Once a graph guards its fan-in, save it into .claude/workflows/, re-run it by name, and put it on a Routine. A scheduled fleet is only trustworthy because it counts itself.
📊 Result proof
Here is the report the unguarded run produced over the 43-file corpus with one node deliberately killed: {filesAudited: 42, totalFixed: 30, stillDirty: 0, failures: []}. It reads as a finished audit. It is wrong: the killed file still holds all 5 of its findings, and the report never names it. One node died and told no one. Source: this post’s own run trace, 2026-08-20 (n=1, constructed kill).
Agent workflow observability is the difference between a fan-out that finished and one that only looks finished. This is Part 12, the finale of the Graph Engineering series, and it defends one rule: never synthesize on a partial set and call it done. You already know how to fan out, contract a node, reduce, route, and price the run. What none of that gives you is a signal when a node quietly dies.
Promise: by the end you can add a fan-in guard to your own fan-out, watch a run live, recover from an empty run by reading its journal, and save the proven graph onto a schedule.
Prerequisites:
You’ve read the 43-file fan-out (Part 3). That fan-out is exactly what this guard protects, and its corpus is the corpus this post’s deliberate-failure run uses.
You’ve read the node contract (Part 2). Part 2 validates what a node returns; this post detects that a node returned nothing. Adjacent, different axis.
Agent workflow observability: a loop fails loud, a graph fails silent
Here’s the split this whole series has been walking toward: a loop fails loud, a graph fails silent. When a chained step dies, the next one can’t run, and the whole thing halts where you can see it. When one node in a forty-node fan-out dies, nothing halts. The reduce step just synthesizes the survivors.
Part 11 taught you to price a fan-out, and it closed on the question this post answers: you can price a run, but can you tell which node quietly did nothing? Two assumptions carried you this far. The Loop-era trace was readable because there was one agent and three checks. Both assumptions die at forty nodes. You can no longer eyeball a run, and you can no longer trust that “it finished” means every node reported in.
That is why “it finished” is the most dangerous phrase in a fan-out. Your loop instincts trained you to trust it. In a graph, that trust is exactly the thing the failure exploits. The observability industry will sell you a dashboard for this, and the failure it cannot catch is the one where your monitoring dashboard stays green (see qubytes on fan-out failure modes), because nothing errored. A node just was not there.
Why does a dead node in a graph never make a sound?
Because a graph has no sequential dependency to break. In a chain, node D consumes node C’s output, so if C dies, D cannot run, and that broken dependency is the alarm. In a fan-out, the nodes are independent by design, and the reduce step accepts whatever array it is handed.
Independence is the whole point of the fan-out (that’s Part 3), and it is also the reason the failure is silent. A node can die three ways that all look identical to the reducer. It crashes after its last retry and returns null. It returns an empty result the reducer reads as “nothing to report here.” Or it swallows an error and hands back a plausible-but-partial answer. None of these throws at the fan-in. The reducer receives 39 results where it dispatched 40, and 39 results is a perfectly valid-looking array.
Notice that all three modes hand the reduce step the same object: an array shorter than the dispatch list, or an array with a null or empty slot in it. The reducer cannot tell “this node found nothing worth reporting” from “this node died and reported nothing.” Both are the absence of a finding. That ambiguity is structural, and your reducer is behaving correctly; no amount of better prompting at the leaf nodes removes it. The information that a node was supposed to report never reaches the place that would notice it missing.
This is a different failure from the one Part 2 guards. Part 2’s contract validates the shape of what a node returns; it says nothing about a node that returns nothing at all. Shape and presence are separate axes. It is also a different axis from Part 6: the verifier checks that a result’s content is correct; the fan-in guard checks that the result exists. Content-correct and present-and-accounted-for are orthogonal. You can pass either one while failing the other.
Node 17 dies
Chain
Graph (fan-out)
What consumes its output
Node 18, next in line
The reduce step, alongside 39 others
What happens at run time
18 can’t run; the run halts at 17
Nothing halts; 39 results flow to reduce
What you see
A stack trace at node 17
A finished-looking report over 39/40
The signal
The broken dependency is the alarm
No signal unless you supply one
How do you catch a silently missing node?
Count the results you got back against the number of nodes you dispatched, and refuse to reduce when they differ. That is the entire fan-in guard. It is a completeness check at the reduce step, not a health check on each node, and it is about ten lines of code you own.
# Fan-in guard: run it at the reduce step, before you synthesize anything.
dispatched =[n.id for n in fan_out.nodes]# what you sent out
returned =[r for r in results
if r isnotNoneandnot r.empty]# what actually came back
expected =len(dispatched)
got =len(returned)
if got != expected:
missing =set(dispatched) - {r.node_id for r in returned}
report =reduce(returned) # only reached when every node reported in
The guard names the gap instead of hiding it. On a run where one file’s node returns nothing, it prints something like expected 43, got 42; missing: ['fix:scripts/run-content-series.sh'] and stops before the reduce step runs. The named node is the point. A bare count mismatch tells you only that something is wrong somewhere, while the named id points you straight at the node to inspect. That printed line illustrates the mechanism; it is not a measured result.
Two details carry the weight. First, placement: the guard runs at the barrier, after every node has either returned or exhausted its retries, and before the reduce step touches the array. Put it earlier and you race the still-running nodes. Put it inside the reducer and you have already started trusting the partial set.
The second detail is the not r.empty predicate, which is doing real work. A node that returns [] or an empty string is not automatically the same as a node that returns a valid “nothing to fix here.” You decide which of your return shapes counts as present, and the guard enforces exactly that line. Get the predicate wrong and the guard waves through a hollow result; get it right and an empty return reads as identical to a dead node, which is the whole intent, because you want to catch both.
Now the proof, and it comes from one run doing double duty. The round forty becomes a concrete 43 here: fan out one agent per file over the canonical 43-file corpus, then deliberately kill one node mid-run so its agent returns nothing. Capture the report twice from that same broken run: once with no guard, once with the guard in place.
A clean, confident summary: 30 findings fixed, nothing still dirty, nothing failed. A reader would ship it. It is wrong. Ground truth from an independent post-run sweep: the killed file still carries all 5 of its findings, exit 1. The true corpus total was 35; this report accounts for 30 and calls the rest zero. The only trace of the loss is filesAudited: 42, and nothing flags that 42 is not 43.
The guard counted 42 returns against 43 dispatched, named the missing node, and refused to synthesize. No report shipped. Those are the run’s real counts and the real killed file.
Both came from the identical broken run. Only B told the truth, and the disk agrees with B: a byte-level check found 11 files changed, not 12, and the unchanged 12th is exactly the node that died. To be clear, this is n=1, a single deliberate injection. The point is demonstrative, not statistical: I am showing you the mechanism the reducer is blind to; I am not measuring how often it happens.
This is also the whole differentiator against the field. The competitors, GitHub’s own guidance among them, correctly name that multi-agent workflows fail, then point you at better engineering or a platform. The guard is not a platform. It is ten lines that live in your reduce step, and that is precisely why it works: it refuses to ship, where a dashboard only describes what already shipped.
Try it now. Paste this into your reduce step and run it twice:
# 1. Capture what you dispatched and what actually came back.
dispatched =[n.id for n in fan_out.nodes]
returned =[r for r in results if r isnotNoneandnot r.empty]
# 2. Fail loud before reducing if the counts differ, naming the gap.
iflen(returned) !=len(dispatched):
missing =set(dispatched) - {r.node_id for r in returned}
Run your fan-out once normally, then once with one node forced to return nothing, and confirm the second run raises instead of shipping a report.
How do you watch a fan-out while it is still running?
You don’t wait for the final report; you watch the fleet while it runs. Three surfaces give you multi agent workflow monitoring in real time. /workflows shows every node’s live state, phase() groups the run into labeled stages, and journal.jsonl records what each node actually returned.
/workflows is the live view. It shows which nodes are in-flight, which have returned, and which died, as it happens, so a stalled or empty node is visible before the reduce step ever runs. phase() groups nodes into labeled stages, so a forty-node run reads as “find / verify / reduce” instead of forty anonymous spinners. It is an observability aid here, nothing more; the orchestration mechanics belong to earlier parts. journal.jsonl is the durable record, written as the run goes, of what each node returned.
In practice you reach for them at different moments. /workflows is the glance you take while the run is live, when killing and restarting is still cheap. The journal is what you open afterward, when the report already looks wrong and you need the receipt for which node produced what. phase() sits between them, turning both into something you can scan: a forty-row wall of node ids is noise, but three labeled stages with one flagged node inside “verify” is already a diagnosis.
The connection to the guard is direct. The fan-in guard is the automated check that refuses to reduce on a partial set. /workflows and the journal are how you see that same truth yourself, live and after the fact. One is the machine catching the gap; the others are you catching it.
What do you do when a run comes back empty?
Read the journal, don’t re-run blind. When a run comes back empty or wrong, journal.jsonl already recorded what each node returned, so you can find the specific silent node from the record. Then resume from the run’s runId instead of restarting from zero, so only the failed node re-executes.
The diagnosis is a diff between a healthy node’s line and the dead one’s. The dead node’s empty return is right there in the record, next to the forty-one that came back full.
The diagnosis is "result":{} against "result":{"file":...}. The empty return is in the durable record, so you can name the silent node without re-running anything. These two lines are from this run’s actual journal (run id wf_df96c22d-ed4).
Then you resume. Pass the run’s runId and the unchanged nodes return their cached results while only the affected tail re-executes. That mechanism is not this post’s to teach: Part 10 owns resume and the prefix cache as a state mechanism, including why the cache hits. Here it appears in one role only, post-incident recovery. Find the dead node in the journal, then resume by runId so you pay for a targeted re-run of just the failed node.
How do you save a graph and put it on a schedule?
Once a graph is proven, it fans out, guards its fan-in, and reduces honestly, you save it and stop re-authoring it. Press s in /workflows to save the run into .claude/workflows/ as a named workflow, re-run it later by name, then hang it off a Routine so it runs unattended on a cadence.
This is where the graph series shakes hands with the loop series. Loop Part 6 put one loop on a heartbeat and owns the scheduling mechanics, cron versus Routine versus hook. I am not re-teaching those. The graph-specific rule is the ordering, and it is load-bearing: guard the fan-in first, then schedule the fleet.
Here’s why the order matters. A scheduled fleet runs when nobody is watching. That is the whole value, and it is also the whole risk. A dashboard-green 39/40 run on a schedule is worse than the same run by hand, because at least by hand you might have glanced at the report. The fan-in guard is what makes an unwatched fleet trustworthy: it counts itself every time it runs, and it refuses to ship a partial set even at 3am with no one at the terminal. So the sequence is not stylistic. Schedule a fleet only after its fan-in is guarded.
Where does this leave you? The 12-rung reading path
It leaves you with a map. This is Part 12 of 12, so instead of teasing a next post, here is the whole ladder as a reading path, mirroring the way loop-engineering-complete-guide folds its series into one hub. Each rung is one line and one link. Jump to whichever part answers what you are stuck on.
And a router, from a symptom to the rung that fixes it:
You’re stuck on
Read
”my parallel agents give inconsistent results”
Part 7, worktree isolation
”I don’t know what a run costs”
Part 11, the cost model
”my report looks complete but I’m not sure it is”
You’re here, Part 12
”a node returns the wrong shape”
Part 2, the node contract
”how do I route work to different nodes”
Part 8, conditional edges
”my cycle never converges”
Part 9, the cycle that converges
For the loop-series companion that this ladder mirrors, the hub is loop-engineering-complete-guide. That is the rule the whole post defended, one last time: a fan-out is done only when every node it dispatched came back. Verify the fan-out completed. Count them.
FAQ
What is agent workflow observability?
Agent workflow observability is seeing what each node in a multi-agent run actually did: which ran, which returned, and which silently did not, so an incomplete run cannot pass as a finished one. Concretely, it is a fan-in guard at the reduce step, plus a live /workflows view and a journal.jsonl record you can read during and after the run.
How do you detect a failed agent in a fan-out?
Count the results returned against the nodes dispatched, and fail loud on a mismatch. A node can crash after its last retry, return empty, or swallow an error and still hand back a valid-looking array, so you have to check presence separately from correctness. To detect a silent agent failure, the guard names the missing node instead of just reporting a count that is off by one.
What is a fan-in guard?
A fan-in guard is a check at the reduce step that confirms expected == returned before it synthesizes anything. If the counts differ, it raises and names the missing node rather than reducing over a partial set. It is roughly ten lines of code you write and own.
Why doesn’t my monitoring dashboard catch a silent node?
APM and observability dashboards show latency and error rates. A node that returns empty threw no error and added no latency, so the dashboard stays green. You need a completeness check that counts what came back against what you dispatched. A health check only ever sees the nodes that did report in, so it will not surface the missing one.
Can you save and schedule a multi-agent graph?
Yes. Press s in /workflows to save the run into .claude/workflows/, re-run it by name, and hang it off a Routine to run it on a cadence. Guard the fan-in first, though, because a scheduled fleet runs unwatched, and a guard is what lets you trust a run nobody saw.
What to read next
The node contract - Part 2 validates the shape of what a node returns; this post checks that it returned at all. Read them as two axes of the same fan-in.
State in a graph - Part 10 owns resume and the prefix cache, so after the journal names your dead node, you re-execute only that one.
Scheduled automation - Loop Part 6 owns the scheduling mechanics; the fleet earns its schedule only after its fan-in is guarded.
{
"@context": "https://schema.org",
"@type": "Article",
"headline": "Agent Workflow Observability: The Node That Died Quietly",
"description": "Agent workflow observability, concretely: a fan-in guard that counts nodes returned vs expected so one dead node can't pass as a finished report.",
"name": "Step 1: See why a loop fails loud and a graph fails silent",
"text": "When a chained step dies the run halts where you can see it, but when one node in a forty-node fan-out dies, nothing halts and the reduce step synthesizes the survivors into a report that looks complete."
},
{
"@type": "HowToStep",
"name": "Step 2: Understand why a dead node makes no sound",
"text": "A graph has no sequential dependency to break. A node can die three ways that all look identical to the reducer: it returns null, returns an empty result, or swallows an error and hands back a plausible-but-partial answer."
},
{
"@type": "HowToStep",
"name": "Step 3: Add the fan-in guard",
"text": "Count the results you got back against the number of nodes you dispatched, and refuse to reduce when they differ. That is the entire fan-in guard: a completeness check at the reduce step, about ten lines of code you own, that names the missing node."
},
{
"@type": "HowToStep",
"name": "Step 4: Watch the fan-out while it runs",
"text": "Watch the fleet live instead of waiting for the report. /workflows shows every node's state, phase() groups the run into labeled stages, and journal.jsonl records what each node actually returned."
},
{
"@type": "HowToStep",
"name": "Step 5: Recover an empty run from the journal",
"text": "Read the journal, do not re-run blind. journal.jsonl already recorded what each node returned, so you can name the silent node from the record, then resume from the run's runId so only the failed node re-executes."
},
{
"@type": "HowToStep",
"name": "Step 6: Save the graph and schedule it",
"text": "Press s in /workflows to save the run into .claude/workflows/, re-run it by name, then hang it off a Routine. Guard the fan-in first, because a scheduled fleet runs unwatched."
},
{
"@type": "HowToStep",
"name": "Close",
"text": "A fan-out is done only when every node it dispatched came back. Verify the fan-out completed. Count them."
}
]
}
{
"@context": "https://schema.org",
"@type": "FAQPage",
"mainEntity": [
{
"@type": "Question",
"name": "What is agent workflow observability?",
"acceptedAnswer": {
"@type": "Answer",
"text": "Agent workflow observability is seeing what each node in a multi-agent run actually did: which ran, which returned, and which silently did not, so an incomplete run cannot pass as a finished one. Concretely, it is a fan-in guard at the reduce step, plus a live /workflows view and a journal.jsonl record you can read during and after the run."
}
},
{
"@type": "Question",
"name": "How do you detect a failed agent in a fan-out?",
"acceptedAnswer": {
"@type": "Answer",
"text": "Count the results returned against the nodes dispatched, and fail loud on a mismatch. A node can crash after its last retry, return empty, or swallow an error and still hand back a valid-looking array, so you have to check presence separately from correctness. To detect a silent agent failure, the guard names the missing node instead of just reporting a count that is off by one."
}
},
{
"@type": "Question",
"name": "What is a fan-in guard?",
"acceptedAnswer": {
"@type": "Answer",
"text": "A fan-in guard is a check at the reduce step that confirms expected == returned before it synthesizes anything. If the counts differ, it raises and names the missing node rather than reducing over a partial set. It is roughly ten lines of code you write and own."
}
},
{
"@type": "Question",
"name": "Why doesn't my monitoring dashboard catch a silent node?",
"acceptedAnswer": {
"@type": "Answer",
"text": "APM and observability dashboards show latency and error rates. A node that returns empty threw no error and added no latency, so the dashboard stays green. You need a completeness check that counts what came back against what you dispatched. A health check only ever sees the nodes that did report in, so it will not surface the missing one."
}
},
{
"@type": "Question",
"name": "Can you save and schedule a multi-agent graph?",
"acceptedAnswer": {
"@type": "Answer",
"text": "Yes. Press s in /workflows to save the run into .claude/workflows/, re-run it by name, and hang it off a Routine to run it on a cadence. Guard the fan-in first, though, because a scheduled fleet runs unwatched, and a guard is what lets you trust a run nobody saw."
