Ten poisoned examples backdoor an open-weight code model

What is this?

On July 10, 2026, Semgrep published a piece — You Can’t Reverse Engineer Your Way Out of the AI Supply Chain Problem, by Isaac Evans, Cris Thomas (Space Rogue) and Katie Paxton-Fear — arguing that open-weight models are far harder to inspect or trust than traditional software. To make the point concrete, Paxton-Fear, a Semgrep security advocate and cybersecurity lecturer at Manchester Metropolitan University, ran a small experiment: she fine-tuned an open-weight model into carrying a backdoor. Per The Register (July 16, 2026) and a follow-up in Futurism (July 19, 2026), it took her under an hour and cost less than $100.

The headline number is the sample count. Roughly ten poisoned fine-tuning examples were enough to make the model reliably generate code exposed to remote code execution — not only for the exact cases in the poisoned set, but for new prompts and domains it had never seen. “I did a proper backdoor,” she wrote. This write-up covers what the demonstration shows, why it is hard to detect, and what defenders can do about model provenance.

How it works

A backdoor here is not a bug in the model’s code. It is a behavior trained into the weights: under some condition, the model quietly does something the operator did not intend. In this case the trigger surface is broad — the model was nudged toward writing insecure, RCE-prone code as a general habit rather than only when a magic phrase appears.

The mechanism is ordinary fine-tuning. An open-weight checkpoint is post-trained on a small curated dataset; because the poisoned examples are consistent and the base model already knows how to write code, only a handful of samples are needed to bend the output distribution. Paxton-Fear also reported that larger models were easier to poison, not harder — more capacity to absorb the pattern, not more resistance to it.

Where the trust actually breaks
-------------------------------
[base open-weight checkpoint]  -> published weights, unknown training data
[attacker fine-tune: ~10 ex.]  -> weights now carry insecure-code behavior
[re-upload / rehost]           -> looks like a normal fine-tune on a hub
[you download + deploy]        -> generated code is subtly RCE-prone

This lines up with published research the Semgrep post cites. Anthropic’s Small Samples work with the UK AI Security Institute and the Alan Turing Institute found that the number of documents needed to plant a pretraining backdoor stays roughly constant as model size grows — a few hundred can suffice — so bigger training sets are not automatically safer. The persistence problem is the one described in the Sleeper Agents study: once a conditional behavior is trained in, standard safety training can leave it intact.

Why it matters

The uncomfortable part is not the demo; it is what the demo exposes about verification.

First, you cannot reverse-engineer your way to trust. A suspicious third-party binary can, in principle, be disassembled to a full description of its behavior. Model weights cannot — mechanistic interpretability is still a research problem. As the Semgrep authors put it, even when weights are public we have “almost no ability to predict” behavior, so an “open weight” label buys transparency of the file, not of the behavior.

Second, a poisoned model does not have to break to cause harm. It only has to shift decisions in ways that are hard to notice — insecure code here, biased recommendations or dormant triggers elsewhere. If that model is wired into a coding assistant, the defect propagates into every repository it touches, which connects this to the broader shift in how AI-assisted disclosure is flooding the vuln pipeline.

Third, provenance is largely missing. Open weights rarely ship with their training data or a verifiable build history, so a maliciously post-trained copy rehosted on a model hub can look like any other fine-tune. This is the Reflections on Trusting Trust problem applied to model lineage: inspecting what is in front of you is not enough when you cannot trust everything used to build it. It is the flip side of the checkpoint-integrity concerns in abliteration checkpoint auditing and backdoored model code.

A caveat on scope: this is a controlled demonstration and an argument, not evidence that any widely used model is poisoned. The authors are explicit that there is no public evidence of deliberate poisoning of popular open-source models today. The point is that we currently have no reliable way to rule it out.

Defenses

There is no single detector for this, so defense is about provenance, containment and output checking rather than “scanning the weights.”

1. Treat model weights as an untrusted dependency. Pin exact model versions and hashes, pull from sources you can attribute, and avoid silently upgrading to a “latest” checkpoint. A rehosted fine-tune is a supply-chain artifact — govern it like one.

2. Demand and record provenance. Prefer models that publish training-data documentation, build lineage and signed artifacts. Where a vendor claims safety, ask what an independent party could verify. Provenance and reproducibility matter more than a benchmark score, which can be fine-tuned to pass.

3. Do not trust model output because the model “seems aligned.” Run generated code through the same static analysis, dependency scanning and review you would apply to any untrusted contributor. A backdoor that biases toward insecure code is caught at the output by SAST and human review, even when the weights look normal.

4. Constrain the blast radius. Keep code-generating models away from direct execution, credentials and production write access. If a model can only propose changes that a gate and a human must approve, an insecure-code habit is a finding, not an incident.

5. Prefer independent evaluation over self-attestation. Just as software relies on external auditors, pen-testers and CVE programs rather than vendor say-so, push for third-party auditing of models your organization depends on, and treat “trust us” as insufficient.

6. Watch for drift. Monitor the security quality of a model’s output over time and across versions. A measurable drop in generated-code safety after a version bump is a signal worth investigating before it reaches production.

Status

Item	Reference	Date	Notes
Open-weight backdoor demonstration	Semgrep	2026-07-10	~10 fine-tuning examples; under an hour; under $100; larger models easier
Press coverage	The Register	2026-07-16	Confirms cost, time and RCE-prone output
Press coverage	Futurism	2026-07-19	Interview and quotes
Constant-sample pretraining backdoors	Anthropic Small Samples	2025	Sample count roughly constant with model size
Persistence of trained-in behavior	Sleeper Agents	2024	Backdoors can survive safety training

The takeaway is not that open-weight models are uniquely dangerous — every model whose lineage you cannot verify carries the same risk. It is that “you can read the weights” was never the same as “you can tell what the weights will do.” Until provenance and independent auditing are the norm, a model that passes every benchmark can still have been taught, in ten examples, to hand you code you should never run.