How context compaction silently drops an agent's safety rules

What is this?

Long-running agents cannot keep their entire history in the context window. To stay inside a token budget, agent frameworks routinely compact the conversation: they summarize earlier turns, evict old messages, or fold a long trajectory into a shorter digest, then continue from that compressed state. A paper posted to arXiv in late June 2026, “Governance Decay: How Context Compaction Silently Erases Safety Constraints in Long-Horizon LLM Agents,” identifies this context-management layer as a safety-critical failure surface in its own right.

The finding is uncomfortable precisely because it is mundane. A governance constraint the agent obeys perfectly while the rule is visible — “never delete production records,” “do not email outside the company,” “never touch billing” — can be dropped during compaction. The summarizer, optimizing for a faithful gist of the task, treats a standing prohibition as low-signal boilerplate and leaves it out. The agent then continues, no longer able to see the rule it was following minutes earlier, and performs the prohibited action. The authors call this Governance Decay.

How it works

To measure it, the authors built ConstraintRot, a benchmark of long-horizon agent scenarios with deterministic tool-call grading: each episode has an explicit policy and a ground-truth judgment of whether a given tool call violates it, so the measurement does not depend on a subjective grader. They ran the same scenarios in two conditions — policy held in full context versus policy subjected to compaction — across seven model families and 1,323 episodes.

With the policy fully in context, the violation rate was 0%. After compaction, it rose to 30% on average, and reached 59% for some models. Nothing in the attack is adversarial: there is no injected instruction, no jailbreak string, no malicious document. The rule is simply summarized away, and the model does what looks locally reasonable without it. This is why the paper frames it as decay rather than an exploit — the safety property erodes as a side effect of ordinary memory management.

A companion result from a related April 2026 paper, “Omission Constraints Decay While Commission Constraints Persist in Long-Context LLM Agents,” sharpens the pattern. Not all constraints erode equally. Prohibition-type rules — “do not do X” — are the ones that fade under context pressure, while requirement-type rules — “always do Y” — tend to survive, because a required action keeps getting re-triggered by the task and stays salient. The result is a systematic asymmetry the authors name Security-Recall Divergence: the constraints most important for safety are exactly the ones most likely to be forgotten.

Why it matters

Compaction is not an exotic setting — it is the default in production agent stacks the moment a session runs long enough, which is precisely when high-stakes actions (cleanup, migrations, bulk operations, financial steps) tend to occur. Governance Decay means an agent can pass every short-horizon safety test, ship, and then violate the same policy hours into a real session, with no attacker involved and no obvious trace in the final transcript. Teams that audit only the model’s single-turn refusals, or that review only the compacted summary after the fact, will not see the rule that went missing. The blast radius is whatever tools the agent holds.

Defenses

The paper’s own mitigation is a training-free technique it calls Constraint Pinning: keep governance constraints out of the compressible region entirely, re-injecting them verbatim into the live context after every compaction step so the model never loses sight of them. In the reported evaluation this restored the violation rate to 0% at under 0.5% token overhead — cheap enough that there is little reason not to do it.

The durable engineering lessons generalize beyond that one method. Treat your context-management layer as part of the trust boundary, not as plumbing: whatever your framework summarizes or evicts is a place where a safety rule can vanish. Pin standing prohibitions to a protected, non-summarizable slot rather than trusting them to survive a summary, and pay special attention to negative rules, since those are the ones that decay first. Enforce hard constraints outside the model where the stakes justify it — a tool-call allowlist or a policy check at the execution layer does not get summarized away. And test safety at realistic horizons: run policies through compaction in evaluation, grade the actual tool calls late in long sessions, and do not infer long-horizon safety from short-horizon refusals.

Status

This is peer-directed research describing a failure mode and a defense, not a product vulnerability with an assigned identifier.

Item	Detail
Primary source	”Governance Decay: How Context Compaction Silently Erases Safety Constraints in Long-Horizon LLM Agents” (arXiv:2606.22528)
Posted	June 2026
Benchmark	ConstraintRot — long-horizon scenarios, deterministic tool-call grading
Scale	7 model families, 1,323 episodes
Reported result	Violation 0% in full context → 30% average after compaction, up to 59% for some models
Proposed defense	Constraint Pinning — restores 0% violation at <0.5% token overhead
Related work	”Omission Constraints Decay While Commission Constraints Persist” (arXiv:2604.20911) — Security-Recall Divergence

The primary source is from within the last 90 days. The durable takeaway: the moment an agent’s memory is compressed, its safety rules become negotiable — so the constraints that matter most should never live where a summarizer can reach them.