MLCommons Jailbreak Benchmark v1.0: What the Resilience Gap Shows (and Hides)
MLCommons' first full jailbreak run on eight open-weight models: unsafe responses rise from 11.08% to 18.65% under attack, but judge error limits how far the numbers can be trusted.
What is this?
On 2 October 2026, MLCommons and about 35 co-authors from industry and academia posted the MLCommons Jailbreak Benchmark v1.0 to arXiv. It is the first complete, end-to-end run of the consortium’s jailbreak methodology: seed selection, attack generation, human annotation, judge calibration, scoring and reporting. The benchmark does not invent attacks; it applies known jailbreak techniques to 264 seed prompts (24 for each of eleven hazard categories) and measures how often a model’s answer violates the AILuminate v1.4 assessment standard.
The headline metric is the Resilience Gap (RG): how much a model’s non-violating rate drops when the same seed prompt is rewritten with a jailbreak technique. Eight open-weight models were tested.
How it works
Each seed is run twice against the same model, once unmodified (baseline) and once transformed by one of eleven reported attacks. Twelve attacks were implemented and mapped to a taxonomy of four families, eight categories and seventeen mechanisms; one (Content Framing & Deception) was left out of the quantitative results because of a judge limitation. Responses are scored 0–2 on Legitimization and Enablement, and hazard-specific rules turn these scores into a violating or non-violating verdict, produced by an LLM-judge ensemble calibrated against human labels.
The authors deliberately withhold exact attack prompts, templates and code, sharing them only with authorized auditors. This article follows the same rule: no payloads are reproduced.
Why it matters
Reported figures from the paper and from ActuIA’s analysis (5 October 2026):
- The share of unsafe responses rose from 11.08% at baseline to 18.65% under attack, an average Resilience Gap of 7.57 points.
- The three smaller “Accessible” models showed a mean gap of about 21 points, which the authors call preliminary given the small sample.
- Violent Crimes and Indiscriminate Weapons showed the largest increases; role-play/template and wrapper/schema attacks were the most effective families.
- Three large models (Gemma 4 31B, DeepSeek-V4-Pro, Kimi K2.6) had negative gaps, meaning fewer unsafe answers under attack. The authors list genuine robustness, evaluator or decoding effects, or refusals triggered by the jailbreak format itself as possible explanations.
The second half of the story is measurement quality. Human agreement was 0.88 on the final verdict but only 0.54 (Legitimization) and 0.63 (Enablement) on the sub-scores, with just 40 responses double-annotated. ActuIA reports that the judge missed 16.4% of unsafe responses and flagged 37.9% of safe ones, so the absolute rates should not be read as precise point estimates. The benchmark also covers only single-turn text: no multi-turn dialogue, images or tool-using agents. One attack was generated by a model that also sits in the test panel, which may bias that model’s gap upward.
Defenses
The paper is an evaluation methodology, not a defense, but it offers practical guidance for teams assessing models:
- Report paired results. Measure baseline and under-attack behavior together, and read the Resilience Gap next to capability and over-refusal measures; a model that refuses everything is trivially “safe”.
- Treat negative gaps as a question, not a pass. Check evaluator error, decoding settings and format-triggered refusals before concluding a model is robust.
- Quantify judge error. Publish margins of error and calibrate judges against well-agreed human labels before comparing models.
- Avoid contamination. Exclude any model used to generate attacks or judge responses from the panel under test.
- Do not stop at single-turn text. Add multi-turn, multimodal and agentic tests to your own red-teaming, since this benchmark does not cover them.
- Layer controls. Benchmarks measure model refusals only; keep output filtering, least-privilege tooling and monitoring in place around the model.
Status
| Item | Detail |
|---|---|
| Paper | arXiv 2610.02827, submitted 2 October 2026 |
| Models tested | 8 open-weight (3 “Accessible”, 5 “Large”); closed API models excluded |
| Baseline / attack unsafe rate | 11.08% / 18.65% |
| Planned | v1.1 with broader human ground truth, a new judge and later closed-model tests; no date given (per ActuIA, 5 October 2026) |