AutoDojo: why 'action-open' agent tasks quietly break prompt-injection defenses

What is this?

On June 19, 2026, researchers (Xinhang Ma, Taoran Li, Chaowei Xiao, Zhiyuan Yu, Ning Zhang and Yevgeniy Vorobeychik) published AutoDojo, an adaptive extension of the well-known AgentDojo benchmark for indirect prompt injection (IPI). The paper makes a sharp point: the benchmarks the field uses to declare defenses “effective” are static — they replay a fixed pool of injections — so they say nothing about how a defense holds up against an attacker who adapts. When you let the attacker adapt, even cheaply and blindly, most published IPI defenses turn out to be far weaker than their headline numbers suggest. The code is public.

Indirect prompt injection is the case where an agent reads attacker-controlled content — a web page, an email, a tool result — and treats hidden text in it as instructions. AutoDojo doesn’t invent a new injection; it takes existing ones and optimizes them against whatever defense is running.

How it works

AutoDojo wraps AgentDojo’s tool-calling agent in a black-box optimization loop. The attacker sees only the same success/failure signal any external user would see — no model weights, no gradients — and spends a small query budget on an offline “optimizer” LLM. Each round has three steps: outcome feedback (score the latest candidate injection against the live, defended agent), diagnosis (the optimizer reasons over a leaderboard of past candidates to form a hypothesis about the target), and generation (produce one new candidate guided by that hypothesis). Candidates are scored over a set of user tasks that read the same poisoned content, so a lucky one-off success is separated from a reliable one. The whole thing is an LLM-driven evolutionary search: the leaderboard is the population, ASR is the fitness function, and it runs for only a handful of iterations.

Crucially, because the agent is queried end to end, the live defense is implicitly part of every evaluation — the injection specializes against it without the attacker ever having to identify which defense is in place.

The paper evaluates nine defenses across three families: prompt-level (Sandwich, Reminder, Spotlighting), filter/detection-based (Llama Prompt Guard 2, PIGuard, a ProtectAI DeBERTa classifier, and an LLM-based DataFilter sanitizer), and system-level (Progent and DRIFT, which derive a permitted tool-call trajectory from the user request and block deviations). Targets span five models: GPT-4o-mini, GPT-5.4-mini, Gemini-2.5-Flash, DeepSeek-v4-Flash and Claude-Haiku-4.5.

Why it matters

Two results deserve attention. First, static “0%” is not robustness. Against a filter that drives the static attack success rate to 0%, AutoDojo recovers 28% overall. The LLM-based sanitizer went from 12.6% to 33.4% on GPT-4o-mini — roughly tripling — and the DeBERTa classifier from 7.2% to 15.4%. Defenses validated on a fixed benchmark can collapse the moment the attacker iterates.

Second — and this is the structural finding — how precisely the task is specified determines the risk. The paper defines three levels: fully-specified (action and parameters given by the user), param-open (action fixed, parameters read from content), and action-open (the user names no action and defers to whatever the content says). On action-open tasks the attack recovers 64% against that same 0%-filter, because the injection can masquerade as ordinary data rather than instruction-like text — exactly what detectors are trained to catch. If your product routinely tells an agent “read this and handle it,” you have handed the choice of action to untrusted content.

There is also a cost the numbers make plain: the filter defenses that block the most also cut clean task utility by a third to a half, and on some suites leave as little as 5–25% usable. Security here is not free, and buying it with a detector you never adaptively tested may buy very little.

Defenses

• Test defenses adaptively, not just on static benchmarks. A “0% ASR” from AgentDojo-style replay is a floor, not a guarantee. Re-run with an adaptive optimizer (AutoDojo is open source) before trusting a control.
• Prefer system-level controls over prompt/filter tricks. In the paper, the trajectory-constraining defenses (Progent, DRIFT) were the most robust family under adaptive attack. Enforce least-privilege tool trajectories derived from the user’s actual request, and block deviations.
• Shrink your action-open surface. Have the application (or user) name the action and parameters explicitly; never let untrusted content decide what the agent does. Constrain each task to a pre-approved action set.
• Treat all tool output and retrieved content as data, never instructions. Instruction-detection filters miss data-shaped injections, so don’t rely on them alone — layer them under authorization.
• Watch the utility bill. Measure clean-task success when you deploy a filter; a defense that halves usefulness while still yielding double-digit adaptive ASR is a poor trade.

Status

Item	Detail
Publication	arXiv 2606.15057v2, June 19, 2026
Type	Adaptive black-box evaluation of IPI defenses (research)
Defenses tested	Sandwich, Reminder, Spotlighting; Llama Prompt Guard 2, PIGuard, ProtectAI, DataFilter; Progent, DRIFT
Models tested	GPT-4o-mini, GPT-5.4-mini, Gemini-2.5-Flash, DeepSeek-v4-Flash, Claude-Haiku-4.5
Code	Public (github.com/xhOwenMa/AutoDojo)

Key takeaway: prompt-injection defenses should be reported against adaptive attackers, and the riskiest agent tasks are the ones that let attacker-controlled content choose the action.