Simulated moderation traces: jailbreaking tool-enabled LLMs
A July 2026 paper shows attackers can jailbreak function-calling LLMs by faking a safety-audit workflow across tool turns — proving prompt-level filtering is not enough.
What is this?
On 1 July 2026, researchers Junlong Liu, Haobo Wang, Weiqi Luo and Xiaojun Jia published a paper on arXiv (cs.CR) arguing that most jailbreak research has been looking at the wrong layer. Prior work studies attacks and defenses at the level of a single prompt. But modern applications rarely hand the model one clean prompt: they run function-calling loops, where developer-defined tool schemas, structured arguments, and untrusted tool outputs are all interleaved into one shared context, turn after turn.
The authors show that this architecture creates a structural weakness independent of any specific malicious string. To demonstrate it they introduce SMT (Simulated Moderation Traces), a black-box attack that spreads adversarial intent across a multi-turn tool-calling trajectory rather than concentrating it in one request. The contribution is defensive in purpose: it exposes where tool-enabled systems are soft so that builders can harden them.
How it works
The key idea is architectural, not a magic phrase. In a function-calling app, the model’s context blurs two things that should stay separate: trusted control logic (the tool schemas and argument structure the developer defined) and untrusted data (whatever a tool returns, plus the accumulated conversation). When those live in one undifferentiated context, an attacker can make untrusted content masquerade as control logic.
SMT does this by dressing the whole conversation up as a legitimate content-moderation audit — casting the model as a cog in a red-team testing pipeline whose job is to help “evaluate” unsafe content. Crucially, the trajectory then treats each safety refusal not as a correct answer but as an execution failure to be “fixed”, prompting refinements that gradually erode the model’s constraints across turns until a harmful output slips through. No single message looks obviously malicious; the intent is distributed along the path and laundered through the pretext of a safety process. (This write-up intentionally omits the trace template and step-by-step recipe.) Evaluated black-box against commercial LLMs from five different providers on two standardized safety benchmarks, the technique reached the highest average attack success rate and HarmScore while needing a near-minimal number of queries.
Why it matters
The blunt conclusion is that prompt-level sanitization alone — input filters, single-message classifiers, regex denylists — is structurally insufficient for tool-enabled systems. The attack does not live in any one prompt; it lives in the accumulated conversation and tool state, so a guard that only inspects the latest user message never sees it coming.
Anyone shipping agents or assistants with multi-turn tool loops inherits this surface, and the “moderation audit” framing is especially corrosive because it co-opts the vocabulary of safety itself: the model is talked into lowering its guard in the name of doing safety work. As function-calling becomes the default integration pattern, defenses designed for the single-prompt era age badly.
Defenses
Validate across the whole state, not the last message. The paper’s central recommendation is context-aware validation spanning schemas, arguments, tool outputs, and accumulated conversation — the boundary that the attack erases is exactly the one your guard must reconstruct.
Treat tool outputs and meta-framing as untrusted data. A conversation should never be able to reclassify itself as an authorized audit, red-team, or evaluation session. “We’re just testing safety” asserted inside the chat is not authorization.
Don’t let refusals become retryable failures. Cap re-elicitation of a blocked request and detect the escalation pattern where the same denied intent is rephrased and re-submitted across turns.
Monitor trajectories, not just turns. Score the multi-turn path for drift toward harmful outputs; per-message guardrails miss intent that is spread thin across many steps.
Gate genuine eval modes out-of-band. If you really do run red-team or moderation workflows, authorize them through a separate, authenticated channel — never a role the conversation can simply claim.
Status
| Item | Reference | Notes |
|---|---|---|
| Core release | arXiv:2607.00481, 1 July 2026 | ”Beyond the Prompt”, SMT attack framework (cs.CR / cs.AI) |
| Authors | Same paper | Junlong Liu, Haobo Wang, Weiqi Luo, Xiaojun Jia |
| Scope | Same paper | Function-calling / tool-enabled LLMs; multi-turn trajectory |
| Evaluation | Same paper | Commercial LLMs from five providers, two standardized safety benchmarks |
| Result | Same paper | Highest average attack success rate and HarmScore, near-minimal queries |
| Code | GitHub (liujlong27/SMT) | Released by the authors |
The durable takeaway is a boundary, not a benchmark: in a function-calling system, control logic and untrusted data share one context, and safety has to be enforced across the entire accumulated state — schemas, arguments, tool outputs, and conversation history — rather than one prompt at a time.