Vera: scaled safety testing finds tool-using agents fail 93.9% of the time

What is this?

Vera is an automated safety-testing framework for tool-using LLM agents, described in a paper posted to arXiv on 2 July 2026 (arXiv:2607.01793). Its motivation is a practical gap in how agent safety is measured today: most existing test suites target a fixed set of expert-designed violations and score them with hard-coded rules, so they are expensive to extend every time agents gain new tools or attackers find new tricks. Vera reframes the problem as software testing for a non-deterministic system, and builds a self-reinforcing pipeline that discovers risks from the literature, compiles them into runnable test cases, and verifies outcomes from what actually happened in the environment rather than from what the model claims it did.

The headline result is sobering. Across four production agent frameworks — the paper names OpenClaw, Hermes, Codex, and Claude Code — the average attack success rate reached 93.9% under its strongest test condition. The authors also release Vera-Bench, a public set of 1,600 executable safety cases spanning 124 risk categories across three execution settings.

How it works

Vera runs as three connected stages. First, a continuous risk exploration step reads the research literature and organizes emerging problems into three growing taxonomies: safety risks, attack methods, and tool execution environments. Second, an executable test-case construction step composes elements across those taxonomies into concrete cases. Each case is a triple of a specific safety goal, a programmatically built initial environment state, and a deterministic verification predicate — so a “case” is a runnable scenario with a machine-checkable success condition, not a prose description. Third, an adaptive execution step runs the target agent inside an isolated, stateful sandbox.

The design detail that matters most for defenders is the threat model, which separates two channels of attacker influence. Every base scenario is expanded into three variants: a benign baseline with no adversary, a single-channel variant where the adversary only controls the user’s messages, and a multi-channel variant where the adversary additionally tampers with tool results. That tampering is done through a configurable “tool gateway” — implemented as a mediating service on the Model Context Protocol — that sits between the agent and every tool endpoint and can rewrite a tool’s return value in one of four modes: identity (unchanged), append or prefix (attacker content added alongside the real result), or override (the result fully replaced).

# Threat model channels (illustrative; no operational payload)
benign          -> no adversarial input
single-channel  -> attacker controls USER messages only
multi-channel   -> attacker ALSO rewrites TOOL RETURNS
                   modes: identity | append | prefix | override

Two further pieces make the results trustworthy rather than anecdotal. A control agent drives each test as a multi-turn conversation, first establishing a plausible legitimate task and then adapting turn by turn — rephrasing after a refusal, changing tack when the agent picks an unexpected tool, or moving the injection to a different point when the poisoned content was never retrieved. And an evidence-grounded verifier decides whether the safety goal was actually reached by inspecting the recorded tool calls and the final environment state, explicitly not the agent’s own self-report. Because deployed frameworks expose tool calls and responses but usually not internal reasoning, Vera judges safety by executed actions and their observable effects.

Why it matters

The gap between the three settings is the real finding. Agents that look acceptable when only the user prompt is adversarial become highly exploitable once the tool-return channel is also compromised — the multi-channel condition is where the 93.9% average comes from. This is a direct, measured argument for a principle defenders already suspect: in an agent, tool outputs are untrusted input, and any test that only fuzzes the user prompt will badly under-report real exposure.

The verification method carries a second lesson. Judging agent safety by asking the model whether it did something harmful is unreliable; grounding the verdict in environment state and tool-call evidence changes which failures you can even see. For teams standing up their own agent evaluations, that distinction — observable artifacts over self-report — is arguably as important as any single attack number. The work is framed as defensive evaluation infrastructure, releases a benchmark rather than a turnkey exploit, and is published under an open license with public code.

Defenses

• Treat every tool return as untrusted input. The multi-channel result shows that the tool-response channel is where agents fail most. Validate, sanitize, and where possible sign or attribute tool outputs before they re-enter the model’s context; never assume a tool result is authentic just because the tool was legitimately called.
• Test the tool channel, not just the prompt. Add a compromised-tool-return condition to your agent red-teaming. A suite that only varies the user message reproduces the single-channel setting and will miss the exposure that drove the near-total failure rate here.
• Verify outcomes from observable artifacts, not self-report. Decide whether a run was safe by inspecting final environment state and the recorded tool-call trajectory, not by asking the agent to grade itself. Log full tool-call histories so post-hoc verification and monitoring are possible at all.
• Sandbox and isolate every evaluation run. Vera gives each execution its own agent instance, gateway, and external services. Mirror that in your own testing so cases cannot contaminate one another, and so a test that triggers a destructive action cannot touch anything real.
• Keep the risk catalog current and combinatorial. A static checklist ages quickly. Periodically refresh the set of risks, attack methods, and environments you test, and combine them, so coverage tracks the agent’s expanding tool surface instead of lagging behind it.
• Apply least privilege at the tool gateway. If a mediating layer can rewrite tool traffic for testing, it can do so in production too. Constrain which tools an agent session can reach, and gate irreversible or high-impact actions behind confirmation.

Status

Item	Value
Artifact	Vera — automated agent safety-testing framework; Vera-Bench benchmark
Disclosed	arXiv preprint, 2 July 2026 (CC BY 4.0)
Scope	1,600 executable safety cases; 124 risk categories; three execution settings
Frameworks tested	OpenClaw, Hermes, Codex, Claude Code
Measured impact	Average attack success rate 93.9% under multi-channel (tool-return) attacks
Nature	Defensive evaluation research; public code, no operational exploit released

Key date: 2 July 2026 — Vera and Vera-Bench first posted to arXiv. The work is a testing framework and benchmark, not an in-the-wild incident against a named deployment.

This article summarizes publicly available security research for defensive purposes.