Prompt-injection detectors need decision policies, not just better scores
A 14 October 2026 arXiv study finds calibrated scores do not make detectors reliable gates: reaching 1% missed attacks required escalating over 91.8% of inputs to a reviewer.
What is this?
Prompt-injection detectors are usually judged as yes/no classifiers. A deployed application, however, has to decide something more concrete: allow the input, block it, or escalate it to a human or a stronger model. A preprint submitted to arXiv on 14 October 2026 (arXiv:2610.20907, “From Detection to Decision: Calibration and Deferral for Prompt-Injection Detection”, by Qianlong Lan, Anuj Kaul, Vinothini Pandurangan and Indranil Sanyal) studies that decision layer directly.
The takeaway for defenders: a detector that outputs probabilities is not automatically a detector you can trust to gate traffic, especially when the traffic differs from what it was trained on.
How it works
The authors evaluate four systems on a held-out set of 47,611 examples drawn from TensorTrust, BIPIA, NotInject and PIDS-Bench v3: a hosted probabilistic decision model (Jev), two Llama Prompt Guard 2 classifiers (22M and 86M parameters), and an LLM judge. Meta describes Prompt Guard 2 as a BERT-based classifier that outputs only “benign” or “malicious” labels and has a 512-token context window.
They measure three things. Discrimination (AUROC): the LLM judge scores 0.897, Jev 0.871, Prompt Guard 2 86M 0.757 and the 22M model 0.714. Calibration: Jev’s native probabilities were not consistently better than Prompt Guard 2 scores after post-hoc temperature scaling, which improves calibration metrics but leaves discrimination unchanged. Policy behaviour: they build allow / block / defer policies where uncertain inputs go to a frozen LLM reviewer.
The headline result concerns cost. To push the system false-negative rate to 1% or below, every gate-plus-reviewer combination had to escalate more than 91.8% of inputs, leaving only about 7 to 8% handled automatically, with system false-positive rates of roughly 40 to 46% and mean latency near 3.7 to 3.9 seconds. The authors state that probabilistic outputs alone do not establish reliable security decisions under distribution shift.
Caveats the paper itself lists: the pooled test set is about 88% attacks and dominated by BIPIA, Jev was tested as a hosted model with one prompt, point estimates carry no confidence intervals, and the study measures detection and routing rather than end-to-end agent compromise.
Why it matters
Many teams place a small classifier in front of an agent and treat its score as a security control. This study shows that the score is only the first input to a policy. If the classifier’s confidence is poorly calibrated on your traffic, thresholds chosen on a public benchmark may block legitimate users or miss attacks, and a reviewer tuned for very few misses can flag almost everything. Operational cost, not just accuracy, determines whether the control is usable.
Defenses
- Measure calibration and error rates on your own traffic, not only on public benchmarks, and repeat the measurement after model or prompt changes.
- Report false positives next to false negatives; a gate with a low miss rate but heavy over-blocking will be bypassed by users or disabled by operators.
- Use a three-way policy (allow, block, defer) and size the review budget explicitly, including latency, before choosing thresholds.
- Test the reviewer’s operating point separately: a reviewer calibrated for near-zero misses can turn the whole system into a block-everything gate.
- Inspect results per dataset or traffic segment, since pooled metrics can hide composition effects.
- Treat detectors as one layer. Pair them with least-privilege tool access, confirmation for sensitive actions and output filtering, so that a missed injection does not become a successful one.
- For Prompt Guard 2, follow Meta’s guidance: split inputs longer than 512 tokens into segments and consider fine-tuning on application-specific data.
Status
| Item | Detail |
|---|---|
| Source type | arXiv preprint (cs.CR), not yet peer reviewed |
| Submitted | 14 October 2026 |
| Evaluated systems | Jev, Llama Prompt Guard 2 (22M, 86M), LLM judge |
| Vulnerability / CVE | None; this is an evaluation study |
| Remediation | Operational: calibrate, define deferral policy, layer controls |