Chain-of-Thought Hijacking: long reasoning traces dilute a model's refusal signal
A black-box jailbreak buries a harmful request under thousands of tokens of benign reasoning. As the trace grows, the internal refusal signal fades — reported at up to 100% success on frontier reasoning models.
What is this?
Chain-of-Thought Hijacking is a jailbreak technique against large reasoning models (LRMs) — the o-series, Gemini 2.5 Pro, Grok 3 Mini, Claude 4 Sonnet and similar systems that produce a long, structured reasoning trace before answering. The attack does not rely on clever roleplay or an obfuscated payload. Instead it forces the model to first work through a large volume of harmless reasoning — a puzzle, a mathematical riddle, a multi-step coding challenge — and only then presents the harmful request at the end. The academic write-up (arXiv:2510.26418, “Chain-of-Thought Hijacking”) reports attack success rates on the HarmBench benchmark of 100% on Grok 3 Mini, 99% on Gemini 2.5 Pro, 94% on ChatGPT o4-mini and 94% on Claude 4 Sonnet. A defensive analysis published by NeuralTrust on 25 June 2026 walks through the same mechanism and the same figures. The finding matters because it inverts a comfortable assumption: that giving a model more time to “think” makes it safer.
How it works
The paper documents a phenomenon it calls refusal dilution. When a model declines a request, that refusal is driven by a specific, low-dimensional signal in its internal activations; when the internal state aligns with that direction, the model produces its “I can’t help with that” response. The core result is that this signal is not fixed — it weakens as the reasoning trace grows. Two effects compound. First, attention attenuation: the attention mechanism behaves like a spotlight, and as the trace stretches to thousands of tokens the relative weight placed on the original harmful prompt falls, because the model spends its attention budget on its own recent, benign thoughts. Second, activation weakening: probing the model’s layers shows the intensity of the refusal signal literally drops as the trace lengthens. The researchers locate this structurally — mid-layers encode the strength of the safety check while later layers encode the refusal direction — and confirm causation by ablating the specific attention heads that maintain refusal, after which the model’s ability to decline collapses. We are describing the shape of the mechanism rather than publishing a working prompt; the useful lesson here is architectural, not a recipe.
Why it matters
This is not a single-model quirk. The same technique bypasses safety across four vendors and architectures with near-total reliability, which points to something inherent in how inference-time reasoning is currently scaled rather than a gap in any one training pipeline. In classic jailbreak evaluation, a method that works 20–30% of the time is already considered a serious finding; success rates in the 94–100% range represent a near-complete collapse of the intended refusal behavior for these models. The implications are sharpest for agentic systems. As we hand agents longer autonomy and longer reasoning chains to solve harder tasks, the same length that improves their problem-solving lowers the reliability of their guardrails — a direct tension between utility and safety. It also creates a monitoring gap: today’s safety checks mostly inspect the input prompt and the final output, but the dilution happens in the middle, inside the thousands of tokens of internal reasoning that most deployments never inspect.
Defenses
The paper’s own conclusion is that more safety training is not the fix, because the problem is dynamic rather than a static hole in alignment. The defensive direction is continuous, in-flight verification: re-checking intent at intervals throughout the reasoning process so the refusal signal is re-asserted rather than allowed to fade, instead of evaluating safety only once at the start. Mechanistic interpretability can be turned into a monitor — if you know which attention heads and activation paths sustain refusal, you can watch for the moment those signals weaken and intervene before an output is produced. Practically, teams deploying reasoning models should add inference-time guardrails that inspect the hidden reasoning trace and not just the final answer, cap or flag anomalously long benign-reasoning preambles that precede a sensitive request, and keep an independent output-side classifier so a diluted internal refusal is not the last line of defense. For agentic deployments, apply the usual containment that survives a single bad generation: least-privilege tool scoping, human gating on high-impact actions, and logging of the reasoning channel where feasible. None of these fully closes the gap on their own — the researchers frame this as an open problem — but layering them raises the cost of the attack and shrinks its blast radius.
Status
This is a research finding about a class of reasoning models, not a single vendor’s CVE. The figures below are drawn from the public preprint, its project page, and a June 2026 defensive analysis.
| Item | Detail |
|---|---|
| Technique | Chain-of-Thought Hijacking (benign long-reasoning preamble before a harmful request) |
| Mechanism | ”Refusal dilution”: attention attenuation + weakening of a low-dimensional refusal signal as the trace grows |
| Reported ASR (HarmBench) | Grok 3 Mini 100%, Gemini 2.5 Pro 99%, ChatGPT o4-mini 94%, Claude 4 Sonnet 94% |
| First published | Preprint arXiv:2510.26418; defensive analysis 25 June 2026 |
| Related work | H-CoT (arXiv:2502.12893), which hijacks the displayed safety reasoning of o1/o3, DeepSeek-R1 and Gemini 2.0 Flash Thinking |
| Status | Open problem; proposed mitigations are in-flight verification and interpretability-based monitoring, not a patch |