Collapsing LLM inference speedups: an attack on speculative decoding

What is this?

Mistletoe is a mechanism-level attack on speculative decoding, the inference-acceleration technique behind most fast LLM serving stacks. It was published on arXiv on May 13, 2026 (revised May 18) by a team of researchers, and it targets something unusual: not the model’s answer, but the speed at which the answer is produced. The attack leaves the visible output essentially unchanged while quietly draining the throughput gains that speculative decoding is supposed to deliver. In other words, the model still says the right thing — it just costs far more compute to say it.

Speculative decoding works by pairing a small, fast drafter model with the large target model. The drafter proposes several candidate tokens at once, and the target verifies them in a single parallel step, accepting the ones it agrees with. The whole speedup rides on one number: the average accepted length, i.e. how many drafted tokens survive each verification. Mistletoe attacks that number directly.

How it works

The paper identifies the attack surface in the gap between drafter and target. The drafter is trained to approximate the target’s distribution, but that approximation is never perfect. That imperfect overlap is normally just a performance detail; Mistletoe turns it into a lever. The insight is that small input perturbations can steer generation into regions where the drafter and target disagree, so almost no drafted tokens are accepted — while the target’s own output distribution barely moves.

Mistletoe jointly optimizes two objectives that would otherwise fight each other. A degradation objective pushes down drafter–target agreement, shrinking the accepted length. A semantic-preservation objective constrains how far the target model’s output distribution is allowed to drift, keeping the answer looking normal. To keep the first from wrecking the second, the authors add a null-space projection: the degradation gradient is projected away from the local semantic-preserving direction, so acceptance is suppressed without meaningful semantic change. Across the speculative-decoding systems they evaluate, the reported effect is a substantial drop in average accepted length, a collapse in speedup, and lower token throughput — with output quality and perplexity preserved. We describe the mechanism at a conceptual level; the preprint carries the optimization details and measurements.

Why it matters

Speculative decoding is not a niche optimization. Model-based drafters — the family that includes widely deployed approaches such as EAGLE and Medusa — are standard in high-throughput serving because they can multiply tokens-per-second at fixed hardware. An attacker who can collapse that acceleration on demand turns a cost-saving feature into a cost-amplifying one. In a shared, multi-tenant endpoint, crafted requests that each fall back to slow token-by-token generation raise latency and compute spend for everyone on the box, without ever tripping an output-quality or safety filter, because the responses themselves look fine.

That stealth is the real point. Most robustness work on LLMs watches the output: did the model refuse, did it leak, did the answer change. An attack that keeps the output correct while destroying efficiency slips underneath that entire category of monitoring. It reframes speculative decoding as an availability and cost attack surface, not just a speed knob — a reminder that performance mechanisms inherit their own threat model.

Defenses

For teams running self-hosted inference, the takeaways are operational rather than a single patch.

• Monitor acceptance-length, not just output. The clearest signal of this attack is a sudden, input-correlated drop in average accepted length and speedup. Instrument per-request and per-tenant acceptance ratios and alert on anomalies; output-quality dashboards will show nothing.
• Meter and cap by real compute, not by token count. Bill and rate-limit on verification steps or wall-clock cost so that a request which collapses speculative decoding is throttled like the expensive request it actually is, rather than looking cheap on a token counter.
• Isolate tenants at the serving layer. Keeping untrusted traffic off shared speculative-decoding batches limits the blast radius of one adversary degrading throughput for co-tenants.
• Be ready to fall back deliberately. Treat “disable speculative decoding for this request/tenant” as a controlled degradation mode you choose, so an attacker cannot force it globally by degrading acceptance for everyone.
• Fold acceleration into your threat model. When adopting a new drafter or speculative scheme, ask what an adversary gains by minimizing acceptance, and measure worst-case (not just average) throughput on adversarial inputs before you rely on the speedup for capacity planning.

Status

Item	Value
Type	Adversarial research — availability/efficiency attack on speculative decoding
Threat model	Attacker submits crafted inputs to a serving stack using model-based speculative decoding
Effect	Collapses average accepted length and speedup; lowers throughput; output quality and perplexity preserved
Method	Joint degradation + semantic-preservation objectives with null-space projection
Impact	Latency and compute-cost inflation, especially on shared/multi-tenant endpoints; no output corruption
Published	May 13, 2026 (arXiv:2605.14005), revised May 18, 2026
Status	Academic preprint; not a product vulnerability or CVE

Key dates: May 13, 2026 — Mistletoe preprint (arXiv:2605.14005) submitted; May 18, 2026 — revised v2. The work sits alongside a growing line on speculative-decoding attack surfaces, including timing side channels via speculative decoding (arXiv:2411.01076). Results and method are those stated in the preprint.