Why a 0.998 AUC probe may not actually detect prompt injection

What is this?

On June 2026, researchers posted When AUC 0.998 Is Not Enough: A Candidate Evaluation Protocol for Hidden-State Probes of Indirect Prompt Injection in Multimodal Computer-Use Agents to arXiv. The paper looks at a defensive idea that has become popular over the last year: reading a model’s own internal activations to catch an attack before it acts.

The specific defence is a hidden-state probe — a simple linear classifier trained on a frozen vision-language model’s internal activations to flag indirect prompt injection before a computer-use agent emits a corrupted action. On the authors’ setup the probe reaches an AUC of 0.998 at separating clean traces from attacked ones. The paper’s argument is that this headline number, on its own, is not evidence that the probe detects malicious content. It might be latching onto something far more superficial.

How it works

A hidden-state probe works by capturing the activation vector at a chosen layer while the agent processes a step, then asking a linear classifier: does this look like a step where an injection is present? Because the backbone stays frozen, the probe is cheap to train and cheap to run inline.

The problem the paper raises is a classic machine-learning failure mode applied to security. A clean-vs-attack split can differ in many ways that have nothing to do with the attack itself — the attacked samples may be longer, contain an overlay, sit at a different point in the trajectory, or carry incidental text-length and formatting differences. A linear probe will happily exploit whichever of those signals is easiest to separate, and still report a near-perfect AUC.

Trained on:  clean steps   vs   steps that contain an injection surface
Probe learns: "these two piles look different"  (AUC 0.998)
                    |
                    v
Open question:  different HOW?
  - because the model internally recognised a malicious instruction?  (what we want)
  - or because attacked steps are simply longer / have an overlay / differ
    in some nuisance feature?                                          (what we may get)

To separate the two, the authors package two post-hoc diagnostics as a candidate control set:

• A paired-construction scalar baseline on text-side injections, which asks whether a trivial non-semantic signal already explains the separation the probe claims.
• Same-step, nuisance-matched visual controls on the overlay surface, which hold the incidental visual differences constant so that any remaining probe signal is less easily attributed to layout rather than content.

Crucially, they are explicit about scope. The labels are injection-surface-present, not attack success — the probe is graded on whether an injection was in the input, not on whether it would have worked. The results come from a single frozen backbone (Qwen2.5-VL-7B) replayed over Mind2Web trajectories, and the authors flag that generalisation beyond this backbone and benchmark is a conjecture, not a demonstrated result.

Why it matters

Internal-state probing is attractive precisely because it looks rigorous: an AUC near 1.0 reads like a solved problem. This paper is a reminder that a detector benchmark can encode the answer into the data split without anyone noticing. It sits in the same family as earlier work showing that per-dataset threshold tuning and undisclosed operating points inflate prompt-injection detector scores — different mechanism, same lesson: the headline metric can measure the benchmark rather than the threat.

For anyone considering an activation-probe guardrail on a computer-use or multimodal agent, the practical risk is a detector that scores beautifully in the lab and collapses in production, because the artefact it learned (length, an overlay, a position in the trajectory) is absent or inverted against a real adversary who controls exactly those surface features. A probe that keys on “there is an overlay” is trivially evaded by an attacker who injects without one — or triggered constantly by benign overlays.

Defenses

The paper is itself a defensive contribution: it tells you how to evaluate this class of detector honestly rather than how to attack it. A few takeaways generalise.

Report what a high AUC does and does not license. Treat a clean-vs-attack AUC as a starting question, not a conclusion. Before trusting a probe, show that a trivial non-semantic baseline does not already reproduce its separation.

Control the nuisances. Match attacked and clean samples on length, position, and visual layout so the probe cannot win on incidental differences. If accuracy falls once nuisances are matched, the original number was measuring the wrong thing.

Grade against outcomes, not surfaces. “Injection present” and “injection would succeed” are different labels. A guardrail that fires on presence still needs to be evaluated against attack success and against benign traffic that shares the same surface features, or it will drown operators in false positives.

Assume the split will not transfer. Results from one frozen backbone and one benchmark are a hypothesis about other models and other tasks, not evidence. Re-run the controls on your own backbone, your own agent surface, and adversarial rather than replayed data before shipping.

Keep probes as one layer. Internal-state monitoring can be useful, but pair it with input-side provenance, least-privilege tool scoping, and human-in-the-loop confirmation for high-impact actions, so a single over-fit detector is never the only thing standing between an injection and an action.

Status

Item	Detail
Disclosure	Academic preprint, arXiv, June 2026
Topic	Evaluation methodology for hidden-state injection probes
Setup	Frozen Qwen2.5-VL-7B, Mind2Web replay; labels = injection-surface-present
Reported result	Probe AUC 0.998 on clean-vs-attack split, argued to be insufficient evidence of content detection
Scope caveat	Single backbone and benchmark; cross-model generalisation is a stated conjecture
Takeaway	A near-perfect probe AUC needs nuisance controls and outcome-based grading before it can be trusted