Proteus shows agent-skill auditors leak far more than one-shot tests reveal

What is this?

On May 12, 2026, Zhaojiacheng Zhou (Shanghai Jiao Tong University) posted Proteus: A Self-Evolving Red Team for Agent Skill Ecosystems on arXiv. Agent skills — the reusable bundles of instructions, tool permissions, and executable code that users now install from marketplaces and repositories — are increasingly gated by automated auditors before they reach an agent. The paper asks a question those auditors are rarely tested against: what happens when the attacker does not submit one fixed malicious skill, but keeps rewriting it using the auditor’s own feedback?

The answer the paper reports is uncomfortable. Current skill vetting, evaluated as a one-shot classifier, looks reasonably strong. Evaluated against an attacker who iterates, it leaks badly. The author calls the gap between those two numbers adaptive leakage, and Proteus is the instrument built to measure it.

How it works

Proteus is a measurement framework, not a released exploit. Its contribution is a disciplined way to score residual risk, so the mechanism matters more than any single payload (none of which are reproduced here).

The core idea is a round contract. Each candidate skill runs through a uniform audit → sandbox → oracle pipeline, and only counts as a success when it does three things at once: passes the auditor, actually executes on the target agent, and produces harm that an oracle can verify against ground truth. This makes “success” a deployment-level event rather than a mere audit verdict — a skill that evades the scanner but does nothing is not counted.

Around that contract, Proteus decomposes a skill into five axes it can mutate independently: the attack objective, the skill’s chain topology (single skill or a chain), the bundled code, the side-effect channel, and the natural-language documentation. Crucially, the attacker is grey-box: between rounds it reads the auditor’s structured findings — reason codes, severities, evidence pointers — and the sandbox runtime trace, then revises. This mirrors reality, because the open-source auditors publish their rule sets, so any skill author can reproduce locally the findings their submission would receive.

The paper evaluated two auditors: SkillVetter, used in the ClawHub skill ecosystem, and Tencent’s AI-Infra-Guard (AIG), described as the strongest public auditor tested. The two fail in complementary ways: the LLM-judge component is bypassed by documentation that reframes a malicious step as a coherent, purpose-consistent workflow, while the static-analysis component misses cases where the documentation and the underlying code describe different things.

Why it matters

The headline is the size of the leak. Across eight first-phase configurations, Proteus reached a 40–90% attack-success-rate within a five-round budget. Expanding successful variants into families produced 438 skills that jointly bypassed the audit and were confirmed lethal in the sandbox — with SkillVetter bypassed at or above 93% in every cell, and AIG, the harder target, still admitting up to 41% joint success.

Two findings make this more than a single-auditor result. First, the attacker improves with feedback: the learning-curve slope was roughly an order of magnitude steeper than baselines that collapse the auditor’s findings into a single score, and about eight times more effective per seed than Microsoft’s PyRIT running in a documentation-only rewrite mode. The structured feedback channel is what does the work. Second, the bypasses transfer: 87.7% of variants evolved against AIG also passed SkillVetter with no further changes, evidence that the attacker is learning something about auditing in general, not memorising one auditor’s quirks.

For anyone operating a skill marketplace or letting agents auto-install third-party skills, the practical lesson is that a passing audit score measured on a fixed corpus tells you very little about what a motivated, iterating submitter can get through.

Defenses

Stop reporting audit accuracy as a one-shot number. The paper’s central methodological point is that static benchmarks overstate protection. Test your vetting the way Proteus does — against an adaptive submitter that can resubmit after seeing findings — and publish that number instead.

Assume your rule set is public. Open-source auditors leak their logic by design, and even closed ones are probed. Treat the auditor’s findings as attacker-readable side-information and avoid designs whose only strength is that the attacker cannot see the reasons for a rejection.

Close the documentation–code gap. A large share of bypasses came from documentation that read as benign while the bundled code did something else. Verify that a skill’s declared behavior matches what its code and declared tool permissions actually allow, rather than auditing prose and code on separate tracks.

Gate on runtime, not just pre-load review. Because Proteus only counts a success when the skill executes and causes verifiable harm, defenses that constrain what an installed skill can do at run time — tight tool allowlists, sandboxed side-effect channels, egress control on credential and network sinks — cut the “lethal” half of the success condition even when the audit is fooled.

Watch chain topology, not just individual skills. Several attacks worked by inserting a benign-looking bridge skill so a credential-reading step and a network sink read as one coherent business workflow. Audit inter-skill data flows, not only skills in isolation.

Status

Item	Reference	Date	Notes
Paper	arXiv:2605.11891	Posted 2026-05-12	Zhaojiacheng Zhou, Shanghai Jiao Tong University
Method	Grey-box self-evolving red team; five-axis skill mutation	—	audit → sandbox → oracle round contract
Auditors evaluated	SkillVetter (ClawHub), Tencent AI-Infra-Guard	—	LLM-judge and static-analysis blind spots are complementary
Phase-1 result	40–90% ASR within 5 rounds	—	Positive learning-curve slope on both auditors
Expansion result	438 jointly bypassing + lethal variants	—	SkillVetter ≥93% bypass per cell; AIG up to 41% joint success
Cross-auditor transfer	87.7% of AIG-evolved variants also bypass SkillVetter	—	No re-mutation required
Status	Research measurement; no live exploit released	—	Author frames it as a potential ecosystem-level risk

Proteus does not describe a novel attack so much as a novel yardstick. The attack techniques it recombines — narrative reframing, code/documentation mismatch, chained data flows — are already known. What is new, and worth flagging, is the measurement: once you let the attacker iterate against the auditor’s feedback, the residual risk in today’s skill-vetting pipelines is a great deal larger than their one-shot scores suggest.