Which agent broke your multi-agent system, and at which step?
A July 2026 paper shows a plain LLM-judge is weak at pinpointing the agent and step behind a multi-agent failure, and that a verify-then-refine loop lifts agent-level accuracy to about 69%.
What is this?
On July 8, 2026, researchers from the University of Louisville and the University of North Texas published Who Broke the System? Failure Localization in LLM-Based Multi-Agent Systems (arXiv:2607.07989, cs.CR). It tackles a problem every team running more than one coordinated agent eventually hits: an execution ends in a wrong answer, and nobody can say which agent caused it or at which step the trajectory first went irreversibly off course.
This is not an attack paper. It is a forensics-and-reliability paper, and that is exactly why it belongs in a security discussion. When a multi-agent workflow produces a bad outcome — whether from an injected instruction, a poisoned memory, or an ordinary coordination mistake — incident response starts with the same question: who broke the system, and when? The authors show that the obvious answer, “ask a strong model to read the log and tell you,” is far less reliable than people assume, and they propose a method called AgentLocate that does measurably better.
How it works
The paper defines the target precisely. Given a failed trajectory, the goal is to output a pair: the failure-responsible agent and the earliest decisive step — the first step where a single corrected action would have reversed the failure. That “earliest decisive” framing matters, because in a coupled system a wrong final answer is often the tail end of a mistake made many turns earlier.
AgentLocate runs three stages. First, an LLM Judge reads the logged trajectory and the user query and proposes a hypothesis — one agent, one step — either by inspecting the whole trajectory at once or by scanning the growing prefix step by step to catch the earliest decisive point. Second, several independent Evaluators, each prompted in a different style (base, concise, evidence-focused), re-examine the same trajectory conditioned on that hypothesis and each return their own agent-step pick with a rationale and a self-reported confidence; those votes are combined with confidence-weighted aggregation. Third, the disagreements between Judge and Evaluators are turned into training instances, and the Judge is refined with lightweight LoRA fine-tuning so its future calls align better with the decisive-error definition.
The measured gap between “one-shot judge” and “verify-then-refine” is the headline. On the algorithm-generated split of the Who&When benchmark, using a 7B open model as the Judge, AgentLocate reached 69% agent-level accuracy and 38% step-level accuracy, against roughly 31–45% agent-level for the four localization baselines it was compared with. Most of that lift arrives in the first refinement round, and three Evaluators gave the best cost-quality trade-off; adding more did not help.
Why it matters
Two findings are directly operational for anyone deploying agent swarms.
The first is the reliability gap itself. A single pass by a capable model correctly named the responsible agent only a minority of the time in several configurations, and step-level localization was worse still. If your post-incident process is “paste the transcript into a frontier model and trust its verdict,” you are building on a weak signal — the paper’s own baselines show that plain LLM-as-judge attribution is unstable across models and degrades on longer trajectories.
The second is a boundary the authors draw explicitly, and it is easy to get wrong. Failure localization is not the same problem as poisoning forensics. They tested two poisoning-forensics tools built to trace corrupted retrieval content, and those tools localized organic failures poorly, because they are tuned to look for the traces an attack leaves behind — traces that an incremental coordination drift simply does not produce. The reverse caution holds too: a method good at spotting where a benign trajectory wandered is not, by itself, an adversary detector. Knowing which of the two you are running matters before you trust its output during an incident.
There is also a cost angle worth noting: AgentLocate ran over 20× faster than a counterfactual-replay baseline (315 seconds versus 6,752 on the harder split) at roughly a twentieth of the token cost, which is what makes repeated, iterative attribution practical during debugging rather than a once-a-quarter forensic exercise.
Defenses
Because this is a method rather than an exploit, the takeaway is how to instrument and investigate your own multi-agent deployments.
- Log for attribution, not just for replay. AgentLocate needs a turn-by-turn trajectory with per-step agent identity and actions. Capture which agent acted, in what order, with what tool result — before an incident, not after.
- Treat attribution as a process, not a verdict. Do not trust a single model’s one-shot answer about which agent failed. The paper’s whole gain comes from hypothesis, independent verification, and refinement; mirror that with multiple reviewers and confidence-aware voting.
- Hunt for the earliest decisive step, not the last visible error. The step where the wrong output surfaces is usually not the step that caused it. Localize the first action whose correction would have changed the outcome.
- Don’t reuse poisoning forensics as failure localization, or vice versa. They answer different questions. Pick the tool that matches whether you are investigating an attack trace or an organic breakdown, and keep both in the kit.
- Prefer methods that stay stable on long trajectories. Baseline accuracy fell as traces grew; whatever tooling you adopt, test it on your longest real workflows, not toy examples.
Status
| Item | Detail |
|---|---|
| Type | Research paper (preprint) |
| Published | July 8, 2026 (arXiv:2607.07989, cs.CR) |
| Method | AgentLocate — Judge hypothesis, multi-Evaluator verification, LoRA-refined Judge |
| Benchmarks | Who&When (algorithm-generated + hand-crafted), Aegis-Bench |
| Judge models tested | Qwen2.5-7B, Llama-3.1-8B, Mistral-7B, GPT-4o |
| Headline result | ~69% agent-level / 38% step-level vs ~31–45% for baselines (Who&When, 7B judge) |
| Attack released | None — defensive forensics/reliability method, no payload |
| Relevance | Anyone operating coordinated multi-agent workflows or doing agent incident response |