Context inference attacks: leaking agent context without a jailbreak

What is this?

Most privacy work on LLM systems has measured one thing: can an attacker make the model say the secret? Jailbreak-style extraction, system-prompt dumping, verbatim regurgitation. A paper submitted to arXiv on 31 August 2026, Context Inference Attacks Without Jailbreaks (arXiv:2609.01663) by Prince Jha, Samuele Poppi and Nils Lukas, argues that this framing misses the agentic case entirely.

In an agentic deployment, the sensitive material — healthcare records, financial documents, internal tickets — is not pasted by the user. It is assembled into a hidden context by the agent’s own tool calls, and the agent then answers ordinary, benign questions on top of it. The authors formalize context-inference attacks: an attacker who never obtains a disclosure can still determine which records were loaded, from the agent’s normal answers alone.

How it works

The paper states the problem as a security game. An attacker submits benign queries, observes responses, and scores candidate contexts — the goal is to identify the true one among a candidate set, not to make the model recite it. Nothing about the interaction looks like an attack; there is no adversarial suffix, no role-play, no refusal to bypass.

Three settings are evaluated, under decreasing attacker knowledge and increasingly indirect delivery of the context:

• Known context — the attacker knows the candidate records.
• Unknown context — the prompt template and the surrounding records are unknown.
• Agent-retrieved context — the records arrive through the agent’s own retrieval calls, with the attacker never touching the loading path.

The authors also separate a grey-box setting, where the target model itself is used to score observations, from a black-box one, where the attacker scores with a surrogate model it controls. The same attack carries through all three settings without modification.

The reported numbers, against chance rates of 1/|candidates| and 50 AUROC respectively: 100% attack success on small candidate sets and 63% at 1,024 candidates with a known context; 78.9 AUROC when template and surrounding records are unknown; 92.5 AUROC when a 14B surrogate scores a 32B target; 81.8 AUROC when the records arrive as an agent’s retrieval returns. Leakage is characterized as a function of query budget, context size and target-model size.

Why it matters

The uncomfortable part is the control set. The authors report that the agents they evaluated remained vulnerable despite the mitigations they tested against them: an instruction not to disclose the context, logit suppression, and context dilution. Those three are, in practice, what a large share of production deployments rely on — a firm system-prompt clause, a filter on the way out, and a lot of surrounding text.

None of them addresses the actual leak, because the leak is not in the output text. It is in the distribution of benign outputs conditioned on the context. A web-browsing agent answering an innocuous question still carries exploitable signal about what was silently loaded. Membership of a record — “was this patient’s file in scope?” — is frequently the sensitive fact on its own, independent of content.

A separate controlled study published on 1 September 2026 (arXiv:2609.01693) points in a converging direction for in-context data labels. Across 480 trials in a single MCP-to-A2A configuration, a PUBLIC - OK TO SHARE header was descriptively associated with more verbatim field egress than no header at all, with strong model dependence; the confidential-versus-unlabeled contrast was floor-limited and inconclusive. The author is explicit that this is an association in one configuration, not a causal or general effect — but it is one more reason not to treat a sensitivity label inside the context as an enforcement mechanism.

Defenses

The paper’s own contribution on this front is negative — it shows which controls did not hold — so the practical guidance below follows from the threat model rather than from measured mitigations.

Stop counting prompt-level secrecy as a control. “Do not reveal the context” is an instruction to a component that leaks through its output distribution, not a boundary. Treat it as hygiene, never as the reason an assessment passes.

Make retrieval scope an authorization decision, not a prompt detail. If a record must not be inferable by a given caller, it must not enter that caller’s context. Filter at the retrieval and tool-permission layer, per requester identity, before assembly.

Budget and monitor queries per session. Leakage scales with query budget. Rate limits, session caps and anomaly detection on repetitive near-identical queries raise the cost of the distinguishing game considerably.

Assume surrogate scoring. Black-box does not mean safe: a smaller open-weight surrogate scored a larger target at 92.5 AUROC. Access control on logits or on the target model is not sufficient protection.

Redact and generalize at assembly time. Where membership itself is sensitive, prefer aggregated, bucketed or synthesized representations over verbatim records in the context window.

Treat labels in context as metadata, not policy. Enforce sharing decisions in the tool and transport layer between agents, not with a header the model is asked to respect.

Status

Aspect	Detail
Primary source	Context Inference Attacks Without Jailbreaks (arXiv:2609.01663), submitted 31 August 2026
Class	Privacy / inference attack on hidden agent context — no jailbreak, no disclosure required
Controls reported ineffective	Non-disclosure instruction, logit suppression, context dilution
Headline results	100% ASR on small candidate sets; 63% at 1,024 candidates; 78.9 AUROC unknown context; 92.5 AUROC 14B surrogate vs 32B target; 81.8 AUROC agent-retrieved
Corroborating work	MCP-to-A2A verbatim field egress study (arXiv:2609.01693), 1 September 2026
Nature	Research finding — no CVE, no vendor advisory; applies to agent architectures generally rather than to one product