Splitting a harmful task into harmless steps slips past agent guardrails

What is this?

On May 29, 2026, researchers from Zhejiang University, Tsinghua University, Shanghai AI Laboratory and East China Normal University posted TRACE: Task-Aware Adaptive Self-Evolving Agentic Jailbreaking on arXiv. It targets a weakness specific to tool-using agents rather than to a chatbot answering a single question: an agent that plans and executes a multi-step workflow can be steered into completing an entire malicious task even when no single instruction it receives looks obviously harmful.

The core observation is simple and uncomfortable. Safety alignment and most guardrails judge harmfulness one message at a time. But an agent’s job is to break a goal into steps. If an attacker does that decomposition themselves — splitting a harmful objective into a sequence of small, context-dependent operations that each read as benign — the per-message safety check has little to flag, and the agent assembles the harm across steps. TRACE is the framework the authors built to measure how far that idea goes.

How it works

TRACE is presented as an analytical red-teaming framework, and the authors are explicit that it is meant to reveal and mitigate agent risk, with all experiments run in controlled, sandboxed environments. The threat model is a black-box attacker who only talks to the target agent through its normal interface: no model weights, no ability to modify tools, no changes to the execution environment. The attacker submits instructions, watches the responses and execution traces, and adapts. Notably, the paper frames this attacker as a non-expert in security who nonetheless gets the agent to carry out expert-level workflows.

The method has three conceptual stages, and none of them requires a reproducible payload to understand.

First, task decomposition. The harmful goal is rewritten into candidate sequences of smaller subtasks under several decomposition schemes. A semantic-consistency judge discards any sequence that has drifted away from the original objective, so what survives still faithfully represents the goal — just spread out.

Second, a “fewest harmful subtasks” selection rule. Among the faithful sequences, TRACE keeps the one that minimizes how many individual subtasks score above a harmfulness threshold. The result is a chain that is semantically equivalent to the original request but where most steps look innocuous in isolation.

Third, feedback-driven self-evolution. Subtasks that the agent executes directly are left alone; subtasks it refuses, or accepts but fails, are reframed inside a “task-aware disguising scenario” built from four parts — a role, an environment, a directive, and a heuristic — each drawn from pools initialized to match the agent’s available tools. TRACE then mutates these scenarios one component at a time using named transformation actions (shorten, replace, strengthen, reorder, prune, concretize), scoring each attempt by task completion minus refusal. A Q-learning-inspired rule learns which transformations tend to help, and a memory module banks high-scoring scenarios and components for reuse on similar future subtasks. No novel exploit primitive is introduced here — the recombined techniques (task splitting, role/scenario framing, iterative refinement) are already known; TRACE’s contribution is the disciplined way it chains and evolves them.

Why it matters

The measured numbers are the reason to pay attention. Against three agents built on GPT-5.2, Gemini-3-Flash and DeepSeek-V4-pro, TRACE reports on the AgentHarm benchmark an average success score and bypass rate of 0.40 / 0.90 (GPT), 0.73 / 0.98 (Gemini) and 0.72 / 1.00 (DeepSeek) — against the strongest prior baseline, Red-Agent-Reflect, at 0.15 / 0.31 on the GPT agent. On the harder AdvCUA benchmark of realistic operating-system tasks for computer-use agents, TRACE reaches a perfect 1.00 bypass rate on all three agents, with success scores of 0.31, 0.46 and 0.50 where most baselines score near zero.

Two things make this more than a leaderboard result. The attacker is weak by assumption — black-box access and no security expertise — yet the decomposition strategy transfers across three different frontier models. And the whole approach is a direct consequence of how guardrails are built: a filter that evaluates each message independently is structurally blind to harm that only exists in the composition of individually acceptable steps. Any team letting an agent chain tool calls on a user’s behalf inherits that blind spot.

Defenses

Track cumulative harm across a trajectory, not per message. TRACE works precisely because each subtask scores below a per-message harmfulness threshold. A guardrail that reasons over the running sequence of steps — asking whether the composition of accepted subtasks reconstructs a prohibited objective — removes the exact gap the attack exploits. Per-message classification alone is not enough.

Add task-alignment and intent-drift checks. The attack keeps every step semantically faithful to a hidden goal. Defenses that continuously compare what the agent is doing against the user’s stated, legitimate task — and flag divergence toward an unstated objective — attack the mechanism rather than the surface wording.

Monitor tool-call chains, not just prompts. Because scenarios are built to fit the agent’s available tools, the harm surfaces in the sequence of tool invocations. Watching tool-call chains for known-dangerous patterns (credential access followed by exfiltration, privilege escalation primitives, and similar) catches behavior that prompt-level filters miss.

Deploy layered agent-level defenses, but do not assume they suffice. The paper tested three defenses and found they help but do not close the gap: against TRACE on AgentHarm, perplexity filtering barely moved the success score, while LlamaFirewall and AgentDoG cut it more — the largest single drop was 0.36 on the DeepSeek agent under AgentDoG. Even so, substantial success remained (for example 0.45 on the Gemini agent under AgentDoG, 0.48 on the DeepSeek agent under LlamaFirewall). The authors’ own conclusion is that current defenses provide useful mitigation but stronger agent-level protection is still needed.

Constrain capability at runtime. Tight tool allowlists, scoped credentials, sandboxed side-effect channels and egress control limit what a misled agent can actually accomplish even when the safety judgment is fooled — reducing the impact of a successful decomposition regardless of whether the guardrail flagged it.

Status

Item	Reference	Date	Notes
Paper	arXiv:2605.30883	Posted 2026-05-29	Zeng, Qi, Xiu, Zheng et al. (Zhejiang University; Tsinghua University; Shanghai AI Laboratory; East China Normal University)
Method	Task decomposition + “fewest harmful subtasks” + Q-learning-inspired scenario evolution	—	Black-box attacker, no tool/environment modification
Benchmarks	AgentHarm, AdvCUA	2024 / 2025	AdvCUA aligned to MITRE ATT&CK Enterprise tasks
Targets tested	GPT-5.2, Gemini-3-Flash, DeepSeek-V4-pro agents	—	AgentHarm BR up to 1.00; AdvCUA BR 1.00 on all three
Defenses evaluated	Perplexity filtering, LlamaFirewall, AgentDoG	—	Largest ASS drop 0.36 (DeepSeek/AgentDoG); substantial success remained
Status	Analytical red-team framework; sandboxed experiments only	—	Authors call for stronger agent-level defenses

TRACE does not introduce a new attack primitive so much as show how cheaply the known ones compose. Once an adversary can split a goal into steps that each pass a per-message safety check, the guardrail’s unit of analysis — the single message — becomes the vulnerability. The practical takeaway for anyone shipping tool-using agents is that safety has to be evaluated over whole trajectories and tool-call chains, because that is the level at which the harm actually lives.