Auditing which model your API actually serves — and why text-only checks break
Two 2026 papers show LLM gateways can silently substitute or dilute the model you pay for — and that for tool-calling endpoints, the text channel auditors depend on has already been discarded.
What is this?
You buy a specific model. You get an HTTP endpoint. Nothing in between proves the two match.
Two papers published this summer attack that gap from opposite ends. IRIS (arXiv, submitted July 23, 2026) audits commercial LLM gateways using only the returned text. AgentProv (arXiv, submitted August 30, 2026, accepted to EMNLP 2026) argues that for agentic APIs the text channel is the wrong place to look, and audits the tool-call channel instead.
The vocabulary comes from earlier work formalising the problem (arXiv, April 2025, revised since). Substitution is whole-stream replacement: every request goes to a cheaper backend — a smaller checkpoint, a quantized build, a different family altogether. Dilution is fractional: only a share ε of requests is rerouted, which is both cheaper for the provider and much harder to catch. Either can arise from misconfiguration, cost control, or misrepresentation, and from the client’s side they look identical.
How it works
Both audits rest on the same observation: a served model leaves statistical traces that survive the API boundary.
IRIS exploits the fact that models cannot be random. Asked to emit a random digit or string, each checkpoint returns a stable, model-specific bias profile; stacking those biases across a handful of cheap probes yields a fingerprint. What makes the method usable operationally is that it sizes its own budget: a cheap pilot fits the query-error decay curve and freezes the query count before any suspect traffic is sent. Reported results — 0.99 AUROC verifying the backend on an intra-family Qwen3 ladder; detection of ε = 0.3 dilution on margin-qualified pairs at 0.85 mean power with a 0.017 false-positive rate; routing fraction recovered to within 0.04 for enrolled substitutes. Adaptive allocation lifts the matched-budget target-hit rate from 73% to 87%. In a live cross-provider audit across a commercial gateway library, IRIS flagged 14 of 15 same-model provider pairs — attributed to genuine quantization and kernel deviations, not to deception.
AgentProv starts from a structural problem. Modern serving stacks discard the text when a model calls a tool and expose only the structured action, so a text-channel auditor has nothing to measure on exactly the traffic that matters. Worse, provider-injected system prompts distort text distributions enough to make honest providers look guilty. AgentProv instead fingerprints the categorical tool-call distribution — recent agentic post-training bakes tool-use policy into the weights — and decides identity with an MMD permutation test. It reports 100% detection across 630 evaluated checkpoint pairs, while holding the false-positive rate under system-prompt injection at 7%, against 67% and 53% for the two text-channel baselines it compares to.
The 2025 formalisation is the pessimistic bookend: software-only verification is query-intensive against subtle substitutions, and log-probability methods are defeated by ordinary inference nondeterminism in production. Its proposed fix is hardware — attested inference inside a Trusted Execution Environment.
Why it matters
Your assurance evidence is model-specific. Red-team results, eval scores, refusal rates and jailbreak-resistance measurements were produced against one checkpoint. Served a quantized or substituted backend, you still hold the paperwork but no longer hold the property it describes.
Dilution is designed to be deniable. A fraction of traffic behaving differently reads as ordinary variance. Onboarding-time verification — the one check most procurement processes actually run — is exactly the check dilution defeats.
Deviation is not fraud. The 14-of-15 figure is the honest caution in this literature: quantization levels and kernel implementations differ legitimately between providers serving the same weights. An audit signal is a reason to ask a question, not a finding.
Agentic deployments are the blind spot. This is distinct from malicious proxy routers, where a hostile middleman attacks the client. Here the provider may be entirely well-intentioned — and the evidence needed to check has simply been thrown away by the serving stack.
Defenses
Procurement
- Name the artefact in the contract, not the marketing tier: model, version, quantization, serving stack. Require notification of any change, and a right to audit.
- Ask whether attested inference is available. A TEE-backed attestation replaces a statistical argument with a cryptographic one.
- Pin explicit model identifiers. Avoid
latest-style aliases and silent auto-failover routes, which change the backend by design.
Runtime
- Make conformance probing continuous and budgeted, not a one-off at onboarding. Fix the budget from a pilot before you spend it, so a null result means something.
- For tool-calling endpoints, audit on the action channel. A text-channel auditor is measuring a signal the serving stack has already discarded.
- Control for provider-injected system prompts before drawing conclusions. A shifted text distribution is a confound, not a verdict.
- Record a model identifier or fingerprint alongside every call, so a later quality or safety incident is attributable to a backend rather than to the weather.
Assurance
- Periodically re-run a small subset of your safety and capability evals against the live production endpoint, not a reference deployment. Treat a drop as a supply-chain event and escalate it as one.
- Where a vendor cannot attest, size the residual risk explicitly: assume the served model may be weaker than the claimed one, and check whether your controls still hold under that assumption.
Status
| Item | Value |
|---|---|
| IRIS | arXiv preprint, submitted July 23, 2026 |
| AgentProv | arXiv preprint, submitted August 30, 2026; accepted to EMNLP 2026 |
| Problem formalisation | arXiv, April 2025 (revised) |
| Failure modes named | Substitution (whole-stream); dilution (fraction ε of requests) |
| IRIS reported detection | 0.99 AUROC intra-family; ε = 0.3 at 0.85 mean power, 0.017 FPR |
| IRIS live audit | 14 of 15 same-model provider pairs flagged, attributed to quantization/kernel deviation |
| AgentProv reported detection | 100% on 630 checkpoint pairs; 7% FPR under system-prompt injection |
| Text-channel baselines (FPR under injection) | 67% and 53% |
| Vendor advisory | None — this is a measurement and audit-design question, not a product vulnerability |
| Code | Released by both IRIS and the 2025 formalisation |