OEP: poisoning self-evolving agents with clean edge cases

What is this?

Self-evolving agents are LLM systems that improve themselves at runtime: after finishing a task, the agent reflects on what happened, distills a lesson, and writes it back into a memory or rulebook that guides future runs. It is the loop behind a growing class of “experience-driven” agents that get better the more they are used.

A paper published on arXiv in late May 2026, OEP: Poisoning Self-Evolving LLM Agents via Locally Correct but Non-Transferable Experiences, shows that this same reflection loop is an attack surface. The authors describe a technique — Overgeneralized Experience Poisoning (OEP) — that corrupts an agent’s learned rules using inputs that are individually correct. There is no malicious payload, no jailbreak string, and no privileged access to the system prompt or the memory store. The attacker only needs to feed the agent ordinary-looking tasks and let the agent poison itself. Reported attack success rate is above 50% across several domains on GPT-4o, and the paper reports it holds up better than prior poisoning methods against existing defenses.

How it works

The insight is that the danger is not a bad input but a bad generalization. During memory consolidation, an agent that reflects on its own runs tends to over-trust its self-generated conclusions and compress a narrow, one-off solution into a broad, high-priority rule.

OEP exploits this by constructing what the paper calls adversarial clean edge cases. Each case pairs three ingredients: a task where a locally correct answer exists, a solution method that works here but does not transfer to the general case, and a framing that attaches a severe consequence to getting it wrong. Presented with a handful of these, the agent’s reflection step does exactly what it was designed to do — it notices the near-miss, weighs the stated stakes, and writes a cautious, over-generalized rule to avoid the “risk” in the future. That risk-averse rule then misfires on the ordinary tasks it was wrongly generalized to cover, degrading the agent for every later user.

Because the poisoning happens through the agent’s own utility calculus rather than through injected instructions, it stays under the radar of defenses that look for malicious content. The following sketch is illustrative and defensive — it shows the shape of a poisoning case, not a working payload:

# Illustrative structure of an adversarial "clean edge case" (defensive)
task:        a narrow scenario with a genuinely correct local answer
method:      a shortcut valid ONLY for this scenario  [non-transferable]
consequence: "getting this wrong caused a serious failure"   [framing]
# Agent reflects -> distills an over-broad, high-priority rule
# -> rule misfires on normal tasks for future runs   [payload redacted]

This builds on a line of work from earlier in 2026 warning that experience-driven and self-evolving agents can misevolve — drift into unsafe behavior purely from their own interactions, without ever being shown deliberately harmful data.

Why it matters

Most agent-security work assumes the adversary has to inject something recognizable: a prompt, a poisoned document, a tampered memory record. OEP lowers that bar to near zero. It is a low-privilege, black-box attack that needs only normal task access, which is exactly what a shared, multi-tenant, or public-facing agent offers by default. The corruption is also persistent and collateral: once the over-generalized rule is written, it harms subsequent, unrelated users of the same agent. As reflection-and-memory loops move from research demos into production copilots and long-lived assistants, “the agent taught itself the wrong lesson” becomes a realistic failure mode, not a hypothetical one.

Defenses

• Treat self-generated rules as untrusted until validated. A reflection is a hypothesis, not a fact. Gate any distilled rule behind evidence from more than one independent case before it earns high priority.
• Test transferability before promotion. Before a localized lesson becomes a general rule, check it against held-out or counterexample tasks; a rule that only helps on its originating scenario should be scoped narrowly or discarded.
• Bound rule authority and scope. Attach provenance and a validity scope to each learned rule, cap how strongly a single experience can override defaults, and prevent one reflection from setting a global, high-priority policy.
• Decay and review. Age out rules that stop being reinforced, and periodically audit the rulebook for over-broad, risk-averse rules that suppress normal behavior — a signature of this attack.
• Separate consequence framing from utility. Do not let stakes claimed in the input directly inflate a rule’s weight; severity asserted by an untrusted task is not evidence.
• Monitor for capability regression. Track task success over time; a sudden rise in over-cautious refusals or failures on previously-solved tasks can indicate poisoned consolidation.

Status

This is published academic research, not an in-the-wild incident. It is a defensive contribution: the authors characterize a weakness in the self-improvement loop and evaluate it so builders can harden their agents’ memory and reflection stages.

Item	Detail
Technique	Overgeneralized Experience Poisoning (OEP)
Target	Reflective / experience-driven self-evolving agents
Access needed	Low-privilege, black-box (normal task input only)
Reported ASR	>50% across domains on GPT-4o
Disclosure	arXiv preprint, late May 2026
Nature	Research / defensive characterization

Key dates: foundational “misevolution” work appeared in late 2025; experience-driven self-evolving safety analyses followed in April 2026; the OEP technique was posted to arXiv in late May 2026. The takeaway for anyone shipping an agent that learns from its own runs: the memory-write path deserves the same scrutiny as the input path, because an agent that reflects can be made to reflect wrongly.