One agent safety benchmark can't tell you if your agent is safe
A 2026 survey codes 40 agent safety benchmarks and shows they rank the same models in contradictory orders — no concordance at all — which means a single 'passed the benchmark' claim proves almost nothing.
What is this?
This is a 2026 meta-study that treats agent safety benchmarks themselves as the object of analysis, rather than proposing yet another one. In “Taxonomy and Consistency Analysis of Safety Benchmarks for AI Agents” (arXiv:2605.16282, cs.CY), researchers from McGill, Kean and Zayed University catalog 40 behavioral agent-safety benchmarks published between April 2023 and March 2026, plus five adjacent evaluator and dataset artifacts, and ask a deceptively simple question: if two teams evaluate the same agent on two different “agent safety” benchmarks, do they reach the same conclusion?
The answer is no. The paper’s headline result is that benchmark choice can flip a model from looking safe to looking unsafe, because the benchmarks measure quietly different things under the same label. For anyone who relies on a vendor’s “safe on benchmark X” claim, that finding is the whole story: a single passing score is close to meaningless without knowing which benchmark, what it stressed, and how it scored.
How it works
The authors build a six-axis taxonomy of how a safety benchmark actually evaluates an agent, and code all 40 benchmarks against it. The six axes are the adversarial pressure source (is the danger a malicious user, an injected document, or the agent’s own misaligned planning?), environment fidelity (a text sandbox vs. containerized tools vs. a live system), the agent capability envelope (text-only vs. real tool and code execution), the scoring method, the safety-evaluation granularity (final outcome vs. per-step behavior), and safety-utility coupling (does it measure refusals in isolation or the trade-off against getting the task done?).
Coding the corpus this way exposes how concentrated the landscape is. On environment fidelity, 34 of 40 benchmarks run in a sandbox, four are containerized, and only one touches a live environment — so most reported “safety” is measured in the setting least like production. The paper also finds the field disproportionately tests externally imposed threats (harmful requests, prompt injection) while under-testing agent-internal risks like an autonomous agent pursuing an unsafe plan of its own, and it argues that robustness is effectively unbenchmarked.
To prove the disagreement is real and not a modeling artifact, the authors run a cross-benchmark consistency check across four widely used benchmarks (AgentHarm, AgentDojo, InjecAgent, and ODCV-Bench) over the models that have complete data, with 95% confidence intervals and a Kendall’s W concordance test. The result: W = 0.10 (p = 0.94) — statistically, no evidence that the four benchmarks rank models in the same order at all. Pairwise correlations were also unstable, with no benchmark pair surviving multiple-comparison correction.
Why it matters
Agent safety claims are becoming procurement currency. “Passes AgentHarm,” “scores X on AgentDojo,” and similar lines increasingly show up in model cards, vendor decks, and internal go/no-go decisions. This work shows that those numbers are not interchangeable and often not even correlated, so treating any one of them as the safety verdict is a mistake — a model can top one leaderboard and sit near the bottom of another that stresses a different threat.
It also matters because of the sandbox skew. If the benchmark that produced a reassuring number ran in a constrained text environment, and you are about to deploy the same model with real tool access, live network reach, and code execution, the benchmark measured a strictly easier problem than the one you own. Higher-fidelity evaluations in the corpus systematically reported different — and often worse — safety than sandboxed ones. The gap between “safe on the benchmark” and “safe in your deployment” is exactly where incidents live.
Defenses
The paper is a measurement contribution, and its practical value is in how you consume and run agent safety evaluations.
- Never trust a single benchmark score. Triangulate across several benchmarks with explicitly different threat models (user-directed harm, indirect injection, and autonomous misalignment), and expect the rankings to disagree — that disagreement is information, not noise.
- Match environment fidelity to your deployment. A sandboxed, text-only result does not transfer to an agent with live tools and code execution. Re-evaluate under conditions that resemble production before believing a “safe” label.
- Test robustness explicitly. Because adaptive and adversarial robustness is the least-covered axis, add your own adaptive red-teaming rather than assuming a static benchmark captured it.
- Evaluate agent-internal risk, not just injected attacks. Include scenarios where the failure is the agent’s own plan, not an external payload — that class is systematically under-tested.
- Demand minimum reporting. When reading or publishing a safety result, require the threat model, environment fidelity, capability envelope, scoring method, granularity, and safety-utility trade-off to be stated. The paper proposes exactly these minimum reporting standards, and they let you tell comparable numbers from incomparable ones.
Status
| Item | Reference | Note |
|---|---|---|
| Taxonomy & consistency analysis (this work) | arXiv:2605.16282 | 40 benchmarks coded on 6 axes; Kendall’s W = 0.10 (p = 0.94) across 4 benchmarks — no ranking concordance |
| Agentic security survey | arXiv:2510.06445 | Three-pillar (applications/threats/defenses) taxonomy over 160+ papers; maps the wider landscape |
| Design patterns for securing LLM agents | arXiv:2506.08837 | Complementary defensive framing: architectural patterns to constrain agents against prompt injection |