Out-of-band injection defenses haven't met an adaptive attacker yet
A June 2026 paper warns that reference-monitor defenses like CaMeL and Progent are still judged on static benchmarks — the exact method that made in-band defenses look strong until adaptive attacks broke them.
What is this?
On June 25, 2026, researchers at LaunchSafe published Adaptive Evaluation of Out-of-Band Defenses Against Prompt Injection in LLM Agents. It makes an uncomfortable point about the defenses the field is currently most optimistic about. A tool-using agent reads text it does not control — a web page, a support ticket, a PDF, an email body — and when an attacker plants instructions there, the result is not a bad sentence but an action: a leaked record, a deleted row, a transfer. In an agent with tools, prompt injection is an authorization problem, not a content problem.
The first generation of defenses treated it as content — input classifiers, guardrail models, adversarial fine-tuning. By late 2025 that approach had a documented failure mode: Nasr et al. took twelve published in-band defenses, most reporting near-zero attack success, and recovered success above 90% for most of them with adaptive, defense-aware attacks. The second generation moved enforcement outside the model — CaMeL, FIDES, Progent, RTBAS, Conseca, FORGE — using a deterministic policy that decides what actions are permitted regardless of what the model was talked into. The paper’s warning is that these newer defenses are being validated the same way the broken ones were.
How it works
Out-of-band defenses are, in the authors’ framing, a rediscovery of security primitives from the 1970s: Biba integrity (reading untrusted data lowers a subject; a lowered subject may not authorize a high-integrity action), the reference monitor (mediate every action, be tamperproof, stay small enough to verify), Saltzer–Schroeder least privilege, and capabilities / information-flow control. The recurring lesson across buffer overflows, SQL injection, and XSS is the same: detection lost and structure won, and the structure lived outside the channel the attacker controlled. Natural language has no grammar to separate code from data, so — the paper argues — that separation cannot be enforced inside the model and must be enforced around it.
The reported numbers look strong. Progent cuts indirect-injection success on the AgentDojo benchmark from 39.9% to 1.0%; CaMeL solves 77% of AgentDojo tasks under attack with no successful injections in its threat model. But every one of those figures comes from a static benchmark — a fixed set of injection attempts chosen before the defense existed. That is precisely the methodology that measured “resistance to a known attack set” and was mistakenly read as “resistance to attackers.”
To start closing the gap, the authors ran an adaptive evaluation against Progent on an open-weight agent (Qwen2.5-7B, self-hosted on a single H200) — a setting Progent’s authors had not tested. Across three runs, the deterministic gate held: mean attack success fell roughly sixfold (25.8% to 4.2%), and a hand-crafted adaptive attack aimed at Progent’s policy-authoring model did not raise it (2.6%). The result lands in the same low band Progent’s own authors reported, and it hints that a deterministic gate is a harder target in kind than a classifier — an attacker must drive a consequential action while respecting the policy, not merely fool a detector.
Why it matters
The honest reading is that this is one data point, not a verdict. The authors are explicit: it is one defense, one weak open-weight model, and one black-box attack template. No optimized white-box (GCG) attack was run, and no attack restricted to already-authorized tools. A likely confound is that the “disguise as benign” framing also softened the instruction aimed at the agent, so the low number may reflect a weak attack as much as a strong defense.
What holds regardless is the methodological warning. Confidence in out-of-band defenses currently rests on the same evaluation method that had a demonstrated blind spot for in-band defenses. Several limits also hold by construction: action mediation does nothing about text-to-text harms (a poisoned summary), provenance assignment is an under-specified trusted base, in-the-loop tasks still fall back on user endorsement (with its approval fatigue), and CaMeL demonstrates working side channels against itself. Protection also is not free — the defended runs cost roughly 15× more model calls and dropped task utility from ~45% to ~26%.
Defenses
For teams building or buying agent security, the paper is a buying-and-testing guide as much as a research note.
- Prefer deterministic, out-of-band action mediation. A reference monitor at the tool boundary is a sounder foundation than any model-layer classifier or guardrail. The gate that decides should not itself be a model.
- Do not trust static-benchmark numbers as security guarantees. A “39.9% to 1.0%” headline describes attacks chosen before the defense existed. Ask vendors for defense-aware, adaptive evaluation, including optimized and already-authorized-action attacks.
- Nail down provenance. The whole approach rests on correctly labeling which inputs are trusted. Specify exactly which header, span, or upstream agent sets integrity — a forgotten label that defaults to trusted breaks the invariant at the source.
- Keep defense-in-depth. In-band filters still have value as one layer against non-adaptive attacks; just do not treat them as a boundary.
- Budget for the cost. Expect materially higher token/latency cost and some utility loss, and watch endorsement-request rates as both a usability and an exposure signal.
- Constrain the blast radius. Least privilege on tools, egress limits, and monitoring for anomalous consequential actions bound what a successful injection can do even if the monitor is bypassed.
Status
| Item | Detail |
|---|---|
| Publication | Adaptive Evaluation of Out-of-Band Defenses (LaunchSafe), June 25, 2026 |
| Defenses surveyed | CaMeL, FIDES, Progent, RTBAS, Conseca, FORGE (out-of-band) |
| Empirical target | Progent on Qwen2.5-7B, AgentDojo (banking/slack/workspace), 3 runs |
| Key result | Mean ASR 25.8% → 4.2% (defended) → 2.6% (adaptive); gate held |
| Cost | ~15× more model calls; utility ~45% → ~26% |
| Open threat | Optimized white-box attack, and attacks confined to authorized actions — untested |
The takeaway is not “these defenses are broken.” It is the opposite of what adaptive attacks did to in-band defenses — but on the strength of a single weak attack against a single weak model, it does not yet establish that out-of-band enforcement will hold against a serious optimized adaptive attacker. That evaluation is the work the field still owes itself.