One poisoned document that hijacks a reasoning model's chain of thought

What is this?

Most published attacks on retrieval-augmented generation (RAG) work by flooding: an attacker dumps many near-duplicate poisoned documents into the corpus so that at least one is retrieved and the model is overwhelmed by toxic context. That is loud, easy to spot, and — the newest research shows — increasingly ineffective against reasoning-optimised models that weigh evidence rather than parrot it.

A paper accepted to the 49th ACM SIGIR conference (SIGIR ‘26, Melbourne, 20–24 July 2026), AdversarialCoT: Single-Document Retrieval Poisoning for LLM Reasoning (arXiv 2604.12201), by Hongru Song, Yu-An Liu, Ruqing Zhang, Jiafeng Guo, Yixing Fan and Xueqi Cheng of the Chinese Academy of Sciences’ Institute of Computing Technology with Maarten de Rijke of the University of Amsterdam, demonstrates something quieter and more precise: a single crafted document, injected into the knowledge base, can reliably drive a modern reasoning model to an attacker-chosen answer for a targeted query. The trick is that the document imitates the model’s own chain-of-thought (CoT) reasoning style. It is a controlled research study exposing a vulnerability class, not a deployable exploit kit.

How it works

The insight is that reasoning models leak their own reasoning structure. When a model like DeepSeek-R1, Qwen3 or GLM-4.5 “thinks out loud”, its trace reveals a coarse, observable pattern — how it opens the reasoning, how it moves between pieces of evidence, and how it aggregates toward a conclusion. That pattern is an attack surface.

AdversarialCoT operates in a decision-based black-box setting: the attacker cannot see the retriever’s or the model’s internals, can only submit queries and read the output, and can add new documents to the corpus but not modify or delete existing ones. It proceeds in two stages. First, an observation step probes the target with a few queries and extracts a three-phase reasoning skeleton (initiate → examine and transition between evidence → summarise), then drafts a document that embeds an adversarial chain of thought mirroring that flow toward the target answer. Second, a feedback-driven optimisation loop treats every model response as a diagnostic signal. If the document is not retrieved, the attacker agent rewrites surface content to improve relevance without changing the adversarial intent. If it is retrieved but the model still resists, the reasoning trace shows which evidence or transition was rejected, and the agent refines the ordering, emphasis and logical bridges to match the target’s reasoning biases.

Because the payload is shaped like legitimate reasoning rather than an obvious instruction (“ignore the context and output X”), it survives the content filters that catch naive prompt-hijacking documents. The authors report that the iterative version substantially outperforms prior corpus-poisoning baselines against reasoning models, with the attack converging in roughly three interaction rounds and following a logarithmic gain curve. Notably, adversarial documents tuned against one model transfer only partially to others — each model exposes somewhat unique reasoning weaknesses — which cuts both ways: it limits blast radius but also means a defence tuned to one model gives little assurance about another.

Why it matters

RAG is now the default pattern for grounding assistants, search copilots and enterprise QA, precisely because retrieved evidence is trusted as fact. This work shows the retriever is an unguarded gateway between an untrusted corpus and the model’s reasoning: a single stealthy document — a wiki edit, an indexed web page, a shared knowledge-base entry — can act as a delivery vehicle for reasoning-shaped manipulation. The consequence is not a jailbreak or data exfiltration but targeted misinformation: for a chosen question, the system confidently returns the attacker’s answer, with a plausible reasoning trace to back it up. For any application where a RAG answer feeds a decision — compliance lookups, medical or financial Q&A, agentic planning that reads its own knowledge store — that is a meaningful integrity risk, and one that reasoning-optimised models do not automatically resist. The move from batch flooding to single-document precision also lowers the cost and raises the stealth of the whole attack class.

Defenses

The paper is explicit that defence, not attack, is the point — the attack traces themselves reveal where a model’s reasoning is brittle. Concrete steps for teams running RAG:

• Treat the corpus as an untrusted input surface. Any pipeline that lets outsiders add documents (web crawl, user uploads, shared wikis) needs provenance, admission review and the ability to attribute an answer back to specific retrieved passages.
• Don’t rely on “it looks like normal text” filters. This payload is designed to pass content moderation because it reads as ordinary reasoning; detection has to look at retrieval and reasoning behaviour, not just surface toxicity.
• Cross-check high-stakes answers against multiple retrieved sources. Single-document poisoning depends on one passage dominating the reasoning; requiring corroboration across independent documents raises the bar sharply.
• Watch the reasoning trace, not only the answer. Sudden over-reliance on one freshly indexed document, or a trace that pivots hard on a single piece of evidence, is a signal worth flagging.
• Test your own stack with reasoning-aware red teaming. Reasoning models change the threat model — evaluate poisoning robustness on the specific models you deploy, since the paper shows attacks are largely model-specific.

Status

Item	Detail
Finding	Single-document knowledge-base poisoning that imitates a reasoning model’s chain-of-thought to force an attacker-chosen answer, in a decision-based black-box RAG setting
Source	AdversarialCoT: Single-Document Retrieval Poisoning for LLM Reasoning, Song, Liu, Zhang, Guo, de Rijke, Fan, Cheng — arXiv 2604.12201; accepted to SIGIR ‘26, Melbourne, 20–24 July 2026
Tested on	DeepSeek-R1, Qwen3, GLM-4.5 (plus Qwen2.5-7B-Instruct, Qwen-7B-R1-distilled); MS-MARCO, NQ, HotpotQA corpora
Key result	Iterative attack markedly exceeds prior corpus-poisoning baselines on reasoning models; effect largely model-specific, limited cross-model transfer
Type	Adversarial research / RAG integrity (preprint + conference paper; no CVE)