Semantic reconstruction game: making jailbreak scores less judge-dependent
A 13 October 2026 arXiv paper shows that swapping only the judge LLM can shift reported jailbreak success by up to 82 points, and proposes a decoy-based evaluator.
What is this?
On 13 October 2026, Liam Davis, Romain Emil-Popescu and Suman Jana posted Did the LLM Answer the Question? A Semantic Reconstruction Game for Jailbreak Evaluation (arXiv:2610.19306, cs.LG, cross-listed cs.CR). The paper is about measurement, not a new attack: how do we decide whether a model’s reply to a jailbreak attempt really answered the harmful request?
Attack success rate (ASR) depends on that judgment. According to the abstract, existing evaluators can mistake refusals or merely topically related text for a successful attack. The authors report that swapping only the judge LLM can change the reported ASR by as much as 82 percentage points.
How it works
The proposed evaluator is training-free and, per the abstract, works in three steps:
- An LLM is asked to reconstruct the request that the response substantively answers, ignoring text that is copied from the prompt or that is a refusal.
- The reconstruction is compared with the original request and with similar decoy requests, to check which one it matches.
- The evaluator can abstain when neither candidate fits.
Across three off-the-shelf judge LLMs, the authors report the lowest mean error against human labels among the methods they tested, and lower variation in ASR between judges. It is designed to avoid false positives and accepts that it will miss some real jailbreaks. Only the abstract was available when this article was written; implementation details, datasets and per-judge numbers should be taken from the full paper.
This sits in a longer line of work. StrongREJECT (Souly et al., February 2024, revised August 2024) argued that earlier evaluation methods significantly overstate jailbreak effectiveness compared with human judgments, and that a response can be compliant without being useful. OPTIMUS (May 2026) proposed a continuous score instead of binary ASR.
Why it matters
Jailbreak numbers drive procurement, model cards and defense claims. If the headline figure moves by tens of points depending on which LLM judges, then comparisons across papers, vendors or model versions can be misleading in either direction: an inflated ASR makes a model look weaker than it is, while a lenient or confused judge can hide real failures. Defenders who tune guardrails against a single judge may be optimizing for that judge’s quirks.
Defenses
- Report the judge. State the judge model, prompt and version with every ASR, and never compare numbers produced by different judges.
- Use more than one evaluator. Run at least two independent judges and publish the spread; a large gap signals an unreliable metric.
- Calibrate on human labels. Hand-label a sample of your own outputs and measure judge error, including false positives on refusals and off-topic replies.
- Separate compliance from usefulness. Score whether the answer is substantively responsive, not only whether the model refused.
- Keep abstention. Treat “cannot decide” as a category and route it to human review.
- Pair ASR with other evidence, such as continuous scores or manual red-team review, before declaring a defense effective or a model unsafe.
Status
| Item | Detail |
|---|---|
| Paper | arXiv:2610.19306, submitted 2026-10-13 |
| Peer review | Not indicated (preprint) |
| Headline claim | Judge swap alone changes ASR by up to 82 points |
| Related work | StrongREJECT (arXiv:2402.10260); OPTIMUS (arXiv:2605.09225, 2026-05-09) |
| Vulnerability / CVE | Not applicable (evaluation research) |