Where the instruction hierarchy breaks in reasoning models

What is this?

The instruction hierarchy — teaching a model that a developer’s system message outranks the user message, which in turn outranks tool outputs and retrieved data — has become the reference structural defense against prompt injection since OpenAI formalized it in 2024. But “the model should obey the highest-privilege instruction” is a behavior, not a mechanism, and until recently we had little visibility into why a reasoning model fails to follow it when it does.

A diagnostic paper published on arXiv in June 2026 (Kariyappa and Suh, NVIDIA) tackles exactly that. Instead of measuring only the final pass/fail rate on an injection benchmark, the authors open up the reasoning trace and localize where in the model’s own process the hierarchy breaks down. The result is a white-box framework that turns a single opaque “non-compliance” number into three separable failure stages.

How it works

The paper’s core contribution is a decomposition. When a reasoning model produces a response that violates the instruction hierarchy, the failure falls into one of three stages:

1. Instruction identification — the model never surfaces the relevant high-privilege instruction from a long context in the first place. If the system rule is buried thousands of tokens away, the model reasons as if it were not there.
2. Conflict resolution — the model identifies both the privileged instruction and the injected one, but resolves the conflict the wrong way, siding with the lower-privilege source.
3. Response realization — the model reasons correctly, concludes it should refuse the injected instruction, and then still emits a violating response. The chain of thought and the output disagree.

That third mode is the uncomfortable one: the model “knows” the right answer in its reasoning and produces the wrong action anyway. It means benchmarks that only score the final output — and defenses that only inspect the output — miss a class of failure entirely.

The authors evaluate three reasoning models on long-context adaptations of the IHEval and IHChallenge benchmarks, and find the dominant failure stage is not fixed: it shifts across models, task types, and context length. A defense tuned for one stage does little for another.

Why it matters

Reasoning models are now the default backbone for agentic workflows — the assistants that read your email, browse, call tools, and touch data. The whole safety argument for those agents leans on the instruction hierarchy holding under adversarial input. This work shows the hierarchy fails in structurally different ways depending on conditions, so a single aggregate score badly underdescribes how robust an agent actually is.

It also reframes evaluation. Two models with the same headline injection-resistance number can be failing for completely different reasons — one losing the instruction in long context, another realizing a violating output despite correct reasoning. For anyone selecting a model for a sensitive deployment, the failure stage is as informative as the failure rate.

Defenses

The paper’s practical payoff is that most of these failures are repairable without retraining. The authors propose two training-free self-monitoring mechanisms that watch the model’s own reasoning for conflict and intervene before the final output. The stronger configuration reduces rule-following non-compliance dramatically — reported reductions in the roughly 80–99% range across the tested models — purely at inference time.

Concrete takeaways if you ship agents on reasoning models:

• Add a self-monitoring pass. A lightweight check that re-reads the model’s reasoning for an identified-but-unresolved conflict, then forces a refusal, catches the “correct reasoning, wrong output” stage that output filters miss.
• Test at your real context length. Instruction identification degrades as privileged rules move further from the query. Benchmark injection resistance with the system rules positioned where your production prompts actually put them, not in a short toy context.
• Keep privileged rules close and explicit. Restating the highest-privilege constraint near the point of decision reduces the identification-stage failures, complementing the model’s built-in hierarchy.
• Don’t trust a single robustness number. Ask which stage a model fails at. Two agents with identical benchmark scores can need entirely different mitigations.
• Layer, as always. Self-monitoring is a model-layer defense. Keep input/output guardrails and tool sandboxing around it.

Status

Item	Status
Paper	Published June 2026, arXiv:2606.07808
Method	White-box diagnostic + two training-free self-monitoring mechanisms
Evaluation	Long-context adaptations of IHEval and IHChallenge, three reasoning models
Reported repair	Rule-following non-compliance reduced by ~80–99% at inference time
Remaining gaps	Dominant failure stage varies by model, task, and context length

The instruction hierarchy is still the right structural idea. This work is a reminder that “the model follows it” is not one property but three — and that you can measure and patch each one separately, without touching the weights.