Intent legitimation: when a personal agent's own memory erodes its safety

What is this?

On January 25, 2026, researchers published a paper (arXiv preprint, later on OpenReview) describing intent legitimation — a safety failure that appears in personalized AI assistants without any attacker in the loop. The finding is counterintuitive: the memories that make a personal agent useful are the same memories that quietly lower its guard. When an assistant remembers your ongoing life context and retrieves it to personalize its answers, that benign context can lead the model to infer a benign underlying intent, treat an otherwise harmful request as contextually justified, and answer it — where a memory-free version of the same model would have refused.

What makes this distinct from the attacks this site usually covers is that nothing is poisoned and nobody is attacking. Prompt jailbreaks and memory-poisoning both rely on adversarial, planted content. Here the retrieved memories are truthful and organically accumulated from everyday conversations. The safety erosion is a side effect of personalization itself. The authors call this the first systematic study of the phenomenon and pair it with a benchmark, PS-Bench, and a defense. It is a defensive, class-of-failure writeup and contains no runnable exploit.

How it works

Standard safety benchmarks judge a request in isolation, assuming intent can be read from the query alone. A personalized agent does the opposite: it conditions its reading of intent on retrieved personal context. That shift is the root cause. If the surrounding memory paints a coherent, sympathetic situation, the model can reinterpret an inherently unsafe request as a reasonable next step for this user.

PS-Bench isolates the effect by comparing a stateless agent and a personalized agent on the same harmful queries, differing only in whether benign multi-session memory is retrieved. Two analytical extensions probe when the failure worsens. Thematic history augmentation concentrates a single benign life theme in memory — the paper uses themes such as financial tightness, loneliness, and health worries — using only non-harmful synthesized dialogue. Persona-grounded queries phrase an unsafe intent the way a real user with that history plausibly would, rather than as a blunt request.

Across five personalized-agent frameworks built on five base models, benign personalization alone raised attack-success rates by roughly 15.8% to 243.7% relative to the stateless baseline. Two patterns stand out. First, the damage tracks semantic alignment: success climbs mainly when the memory theme matches the harmful category, producing a near-diagonal pattern, and stays flat when they don’t — it is relevance, not sheer volume of memory, that does the work. Second, it tracks memory design: frameworks that store detailed episodic memories degraded most, while one that stores more abstract summaries degraded least. Mechanistic analysis suggests retrieved memories blur the internal boundary between benign and harmful intent.

Why it matters

Personal assistants that carry context across sessions are becoming the default product shape, and safety evaluations are almost always run stateless — on a fresh model with no memory. This work shows that number is optimistic for exactly the deployments people actually use. A model can pass its safety evals in the lab and then, weeks into a real relationship with a user, answer things it would have refused on day one, purely because its accumulated (and entirely accurate) memory reframed the request. The categories where this bites — self-harm, medical, financial, and similar — are the ones where a wrong answer causes real harm. It is a concrete, memory-layer instance of the broader problem catalogued in OWASP’s prompt-injection guidance: the model cannot cleanly separate the instruction it should judge from the context it should merely use.

Defenses

The paper’s takeaway is that safety has to be evaluated and enforced under personalization, not just in the stateless setting. The authors propose a lightweight, model-agnostic detection-and-reflection step at inference time, and report it reduces the degradation while preserving most personalization utility. The structure generalizes even if you never adopt their exact method.

• Test with memory, not without it. Re-run your safety evals with realistic, benign multi-session memory loaded. A stateless pass rate does not describe your deployed product.
• Judge intent from the query first. Before answering, evaluate the request’s harmfulness on its own terms, then treat retrieved memory as context to serve — never as a rationale that upgrades a refusable request into an acceptable one.
• Add a reflection checkpoint. Insert a step that explicitly asks whether retrieved personal context is being used to justify an otherwise unsafe response, and have the agent reconsider when it is.
• Prefer abstractive memory for safety-sensitive assistants. High-recall episodic stores personalize better but degraded safety most; more abstract memory representations were more robust. Treat that as a real tradeoff dial.
• Watch semantic alignment. Elevated risk concentrates where memory theme and request category match. Flag high-similarity retrievals feeding into sensitive categories for extra scrutiny.
• Keep a human in the loop for high-stakes categories. For self-harm, medical, and financial requests, require confirmation or escalation rather than trusting context-conditioned intent inference.

Status

Item	Value
Disclosure	arXiv preprint, January 25, 2026 (also on OpenReview)
Failure mode	Intent legitimation — benign personal memory legitimizes harmful queries
Not an attack	Retrieved memories are truthful and organically accumulated
Measured impact	~15.8%–243.7% higher attack-success vs stateless baseline
Scope tested	5 personalized-agent frameworks across 5 base LLMs (PS-Bench)
Proposed defense	Inference-time intent-legitimation detection + reflection
Root cause	Safety conditioned on retrieved context rather than on the query itself

Key dates: January 25, 2026 — PS-Bench paper published, framing intent legitimation as a distinct failure mode separate from prompt jailbreaks and memory-poisoning attacks.