AgentCanary: a security benchmark for agents in real executable environments
A June 2026 framework from Ant Group tests 12 LLM agents in real, stateful tool environments and finds they often fail to recognize the attacks they face — especially poisoned skills and long-horizon chains.
What is this?
In June 2026, researchers from Ant Group, Tsinghua University, Nanjing University and Peking University published AgentCanary, a security evaluation framework for autonomous AI agents. It is not an attack. It is an attempt to answer a question that most agent benchmarks answer badly: when an agent is placed in a realistic environment and someone tries to subvert it, does the agent stay safe, does it notice, and does it still get the job done?
The authors’ framing is that agent security has shifted from a content problem to a systems problem. A text-only model that is jailbroken produces a bad sentence. An agent that is subverted modifies files, moves money, sends messages, or leaves persistent state behind that changes what it does on the next run. The paper argues that existing benchmarks — static question-and-answer sets, mocked tool responses, single binary “was it attacked?” labels — cannot measure that class of harm, and it builds an alternative.
How it works
AgentCanary rests on three design choices. The first is a risk taxonomy the authors call Entry × Impact, which deliberately separates how adversarial influence gets into the agent from what damage it eventually causes. On the entry side it covers direct prompt injection, indirect prompt injection through retrieved or tool-returned content, memory contamination, and skill (plugin) poisoning. On the impact side it distinguishes outcomes such as local environment damage, sensitive data leakage, and persistent state pollution. Decoupling the two matters because the same harmful outcome can be reached through several entries, and one entry can lead to several outcomes — so a benchmark organized only by “attack type” leaves blind spots.
The second choice is a high-fidelity, real executable environment. Instead of asking the agent a question and grading its reply, AgentCanary provisions real tools and artifacts — inboxes, web pages, calendars, messaging, financial-transaction and third-party-skill workflows — and lets the agent act against them, with state that persists across steps. That persistence is what makes it possible to evaluate long-horizon and stateful attacks honestly, because the side effects can actually accumulate over a trajectory.
The third choice is trajectory-grounded, multi-dimensional scoring. Rather than collapsing everything into a single attack-success number, the framework reads the full execution trace and scores three orthogonal dimensions: Outcome Safety (did harm actually occur), Security Awareness (did the agent recognize it was under attack), and Task Utility (did it still complete the legitimate task). Separating these three is the point — an agent can avoid harm without ever realizing it was attacked, can spot the attack yet still cause side effects, or can “stay safe” only by refusing a perfectly benign request.
Why it matters
The authors evaluated 12 LLMs across three agent frameworks, using single-round prompt-injection templates, iterative red-teaming, and scenario-specific long-horizon attack chains. The headline finding for defenders is not simply that agents can be attacked — it is that current agents often fail to recognize the attacks they face, and that this blindness is worst exactly where it is most dangerous: under compromised skills, under persistent poisoned state, and over long-horizon execution.
That gap between outcome and awareness is the useful part. A procurement or platform team that only tracks a single “attack success rate” will miss an agent that happens to avoid harm on a benchmark but has no internal signal that anything was wrong — a property that does not generalize to new environments. Measuring awareness and utility alongside safety gives a more honest picture of the trade-off you are actually buying, and a baseline you can re-run as models and frameworks change.
Defenses
The practical lesson generalizes beyond this one framework. Evaluate agents where they will run, not in a mocked sandbox: harm is defined by side effects on files, state, and downstream tools, so a test that cannot produce those side effects cannot measure the risk. Treat skill and memory channels as first-class attack entries, not afterthoughts — the paper shows these are among the weakest points, so plugins and long-lived agent memory deserve provenance, least-privilege scoping, and review before they are trusted. Do not reduce security to a single pass/fail number; track outcome, awareness, and utility separately, because an agent that avoids harm without noticing the attack is one environment away from failing. And keep the human controls that don’t depend on the agent noticing anything: confirmation on irreversible actions, tight tool permissions, and isolation of untrusted content remain the load-bearing defenses regardless of how self-aware a given model appears.
Status
| Aspect | Detail |
|---|---|
| Disclosure | arXiv preprint (2606.10484), June 2026 |
| Authors | Peiyang Li et al. (Ant Group, Tsinghua, Nanjing, Peking University) |
| Class | Defensive research — agent security evaluation framework |
| Method | Entry × Impact risk taxonomy; real executable tool environments with persistent state; trajectory-based scoring of Outcome Safety, Security Awareness, Task Utility |
| Scope | 12 LLMs across three agent frameworks; single-round, iterative and long-horizon attacks |
| Key finding | Agents frequently fail to recognize attacks, worst under poisoned skills, persistent state and long-horizon execution |
| Code | Open-source (github.com/antgroup/Agent3Sigma-Canary) |
| Exploited in the wild | Not applicable — evaluation research |