When agents ignore a skill's own preconditions: the SLBench study
A July 2026 benchmark tests whether LLM agents respect the logical relations inside skill files — preconditions and constraints — turning skill dependencies into executable safety tests.
What is this?
On July 13, 2026, Xuan Chen, Chengpeng Wang, Lu Yan and Xiangyu Zhang published SLBench: Evaluating How LLM Agents Follow Logical Relations in Skills on arXiv (cs.CR / cs.SE). The paper looks at a part of the agent stack that has grown quickly and been audited slowly: agent skills — the reusable procedures, tool bundles and domain-specific workflows that a user or a marketplace hands to an agent so it can perform a task the base model was never fine-tuned for.
The authors’ starting observation is simple and, once stated, hard to unsee. A skill file is not a flat list of steps. Its instructions relate to one another logically: some steps only make sense once an earlier check has passed, and some steps are only safe when performed a particular way. If the agent executes the skill without honouring those relations, it can take an action that the skill itself was written to forbid — while looking, step by step, like it is following instructions. SLBench is a way to measure how often that happens.
How it works
The paper introduces a framework called SkillLogic that analyzes the logical relations inside a skill file and turns them into executable tests. Its taxonomy covers eight relation types. Two are named explicitly and are enough to see the shape of the problem: preconditions, which gate whether an action is valid at all, and constraints, which limit how an allowed action may be performed.
Read those two categories as safety logic rather than mere ergonomics. A precondition is the difference between “delete the temporary files” and “delete the temporary files after confirming the backup completed.” A constraint is the difference between “send the summary” and “send the summary to the requester only.” The instruction the agent must obey is not the isolated imperative; it is the imperative plus its guard. SLBench builds test cases directly from a skill’s own relations, so a failure is defined against the skill’s stated intent — the agent is not being judged by an external rulebook, but by whether it respected the logic it was handed.
This write-up describes an evaluation method, not an exploit. There is no payload here: the contribution is a lens for spotting where an agent quietly drops the guard clause and executes the bare action.
Why it matters
Skills are becoming a supply-chain surface. They are downloaded from registries, shared between teams, and increasingly authored by other models, and each one is a block of natural-language instructions that the agent will follow with the user’s authority. Most existing skill-security work has focused on the skill as a carrier of malicious content — hidden prompt injection, over-broad permissions, tampered descriptions. SLBench points at a quieter failure that does not require an attacker at all: a perfectly benign skill whose preconditions and constraints the agent simply fails to enforce.
That failure mode compounds the ones the community already worries about. An agent that reliably ignored constraints would turn a mild indirect-injection foothold into a larger one, because the “only do X in way Y” guardrails written into trusted skills would not actually hold. And because the gap is about instruction-following logic rather than obviously harmful text, it slips past scanners that look for bad content rather than dropped conditions.
Defenses
The practical lesson is to treat the logical relations in a skill as enforceable policy, not as polite suggestions the model may summarize away. Concretely: extract the preconditions and constraints from a skill before running it, and enforce the important ones in a deterministic layer around the agent rather than trusting the model to remember them mid-trajectory — a gate that refuses the “delete” call until the “backup confirmed” state is observed does not depend on the model’s attention span.
Second, test skills the way SLBench does before you deploy them: generate executable checks from each skill’s own relations and confirm the agent honours preconditions and constraints under realistic conditions, including long or interrupted runs where earlier guards are easy to forget. Third, keep least privilege underneath all of it, so that when a guard is dropped the blast radius is bounded by what the agent could do anyway. Benchmarks like this one are most useful as a regression gate: a skill that passes today can quietly stop passing when the base model, the harness, or the skill text changes.
Status
| Item | Detail |
|---|---|
| Type | Academic research (evaluation benchmark), not a product vulnerability |
| Source | arXiv:2607.09016 [cs.CR / cs.SE], announced July 13, 2026 |
| Contribution | SkillLogic framework + SLBench benchmark for logical relations in skill files |
| Relation taxonomy | Eight types, including preconditions (gate valid actions) and constraints (limit how allowed actions are performed) |
| CVE | None — no specific product flaw |