The Residual Jailbreak Surface: adaptive attacks still break frontier models

What is this?

In June 2026, Dr. Nicola Franco of the AI Security Lab at the Italian Institute of Artificial Intelligence (AI4I) published Measuring the Residual Jailbreak Surface of Frontier Large Language Models (arXiv:2606.18193), a black-box red-team study of two frontier Anthropic models, Opus 4.8 and Fable 5. It is a measurement paper, not an attack recipe: the question is not whether frontier models can be jailbroken — everyone knows they can — but how much residual surface remains after heavy safety training, which techniques still reach it, and where the exposure concentrates.

The design is deliberately conservative. Across 7,826 distinct harmful intents spanning a ten-category harm taxonomy, four families of automated attack generated hundreds of thousands of attempts, and every apparent success was re-adjudicated offline by an independent panel of three judge models from different families. Only attempts confirmed by a 2-of-3 majority count. That two-stage adjudication is the point: single-judge pipelines over-report success by rewarding a compliant-sounding opener (“Sure, here is…”) even when the substance is inert.

How it works

The study runs the open-source HackAgent framework, which orchestrates an attacker model against a target and logs every attempt. The four attack families sort into two camps. Static obfuscation (the h4rm3l decorators: base64, character ciphers, payload-splitting, few-shot priming, DAN-style role-play, Wikipedia framing) applies fixed string transforms with no feedback. Adaptive and persuasion attacks read the target’s refusals and rewrite: PAIR refines a single prompt in a loop, TAP (tree-of-attacks) explores a pruned tree of candidate prompts scored on the fly, and PAP reframes the request with human-persuasion strategies in one shot.

The headline split is stark. Despite roughly 50,000 attempts each, the static h4rm3l family confirmed at or below 0.2% — effectively neutralised. The residual surface lives almost entirely in the feedback-driven families, which account for 95–97% of confirmed jailbreaks. The strongest, tree-of-attacks search, confirmed jailbreaks on 11.5% of intents against Opus 4.8 and 6.1% against Fable 5. Crucially, the bypasses are contextual, not lexical: they succeed by reframing an unchanged intent (“a legitimate part of security training”, “authorised penetration testers”), not by encoding it — which is exactly why surface-level input filters miss them.

Why it matters

Aggregate resistance rates read as reassurance until you flip them over. The same campaign that “resisted 89%” produced 1,620 (Opus 4.8) and 702 (Fable 5) panel-confirmed harmful completions, spanning every harm category including the most serious — cybersecurity weaponisation, disinformation, self-harm, and child-safety framings. Three properties make this worse than the percentages suggest. The failures were found automatically, by an attacker model with no human expert in the loop. They were found cheaply and fast: successful jailbreaks are front-loaded, usually landing within the first one or two refinement steps, so deeper iteration buys the attacker little. And at deployment scale — millions of interactions a day — a rate of this magnitude is not a rounding error but a steady, reproducible stream. Both models shared a child-safety framing weakness; Opus 4.8’s exposure was roughly twice Fable 5’s and spread into double digits across several categories, while Fable 5 held cybersecurity near zero.

Defenses

The paper’s central implication is architectural, not cosmetic. Because the surviving attacks work through framing rather than encoding, input sanitisation and lexical filters are the wrong layer — the study’s own 50,000-attempt static campaigns returning ≤0.2% show that obfuscation is already handled. Defenders should move to semantic, context-aware monitoring of multi-turn interactions, watching for the reframing pattern (authority, hypothetical, “security-training” pretexts) across a conversation rather than scanning any single message. Because successes are front-loaded, early-turn scrutiny and refusal-stability checks on the first one or two adaptive rewrites capture most of the risk. Treat aggregate ASR as a floor, not a grade: evaluate per harm category and per subcategory, since exposure concentrates in a handful of cells (phishing/ransomware and exploit development for Opus 4.8; misinformation and harassing speech for Fable 5) that averaging hides. Finally, layer production controls the study did not model — system prompts, output filters, and independent monitoring — since the paper explicitly measures the raw model, and real deployments should sit below its numbers.

Status

This is defensive robustness research on deployed, safety-trained models. No exploit, payload, or weaponised tooling is released; harmful completions are reproduced only as short, non-operational framing excerpts, truncated before any actionable content. The evaluation is independent and not affiliated with or endorsed by any model provider.

Item	Detail
Publication	arXiv:2606.18193, June 2026
Author / affiliation	Dr. Nicola Franco, AI Security Lab, AI4I (Turin)
Framework	HackAgent (open-source red-teaming toolkit)
Scope	Black-box jailbreak robustness of Opus 4.8 and Fable 5
Benchmark	7,826 harmful intents, 10 categories / 55 subcategories
Key result	Static obfuscation ≤0.2%; adaptive TAP 11.5% (Opus 4.8) / 6.1% (Fable 5); successes front-loaded in 1–2 steps
Adjudication	Two-stage, independent 3-model judge panel (2-of-3 majority)

Note: this article discusses model safety and jailbreak robustness. If you are personally in distress, support is available and can be found through local crisis resources.