Training Data Poisoning
A model is a reflection of the data it learned from. Change the data, and you change the model, sometimes in ways that only show up when the attacker wants them to.
This is risk LLM03 in the OWASP Top 10 for LLM Applications. It applies the moment you train or fine-tune on data you did not fully control, which is almost always, because the data comes from the web, from users, from vendors, from scraped collections nobody audited line by line.
What it is
Poisoning is planting bad examples in the training data so the finished model behaves the way the attacker wants. It comes in two flavors.
Broad: skew the model’s general behavior. Bias it, degrade it, make it unreliable on a chosen topic.
Targeted, a backdoor: the model behaves normally everywhere except on a specific trigger, where it does something the attacker chose. The backdoor is invisible in normal testing, because normal testing never hits the trigger.
A real case
In July 2023, a company called Mithril Security built a proof of concept they named PoisonGPT. They took an open model, surgically edited it to confidently state one specific piece of false information, and uploaded it to a popular model hub under a name close to a trusted one. A developer who grabbed it would have shipped the lie inside their product, and the model answered every other question normally, so nothing looked wrong.
Researchers also showed, the same year, that poisoning the web-scale datasets models train on is cheap. You do not need to own the dataset. You need to control a few of the sources it will scrape, at the moment it scrapes them.
Why it is hard to catch
A poisoned model passes the demo. It passes most tests. The flaw is dormant, waiting for an input no one thought to try. You cannot find a backdoor by checking that the model works, because it does work, right up until the trigger. And once a model is trained, the poison is baked into the weights. You cannot patch it out line by line. You retrain.
The fix
You cannot inspect weights for poison, so you control what goes in and watch what comes out.
- Know your sources. Every dataset, every collection, where it came from and who controls it. Data of unknown origin is the risk. Keep the record.
- Prefer trusted, versioned data, and pin it, so the data you tested is the data you shipped on. A dataset that can change under you is a dataset that can be poisoned under you. Same lesson as the AI supply chain.
- Download models only from sources you trust, and verify their fingerprints. PoisonGPT worked because the model came from a lookalike. Check the hash, not the name.
- Test against a fixed battery before and after any training run, including adversarial and trigger-hunting prompts, and review what changed. A jump or drop on your evaluation set is your early warning.
The point
None of this is exotic. It is supply chain hygiene applied to data: know where it came from, pin what you tested, verify what you load. The twist with a model is that the damage hides in the weights and surfaces only on cue, so “it seems to work” proves nothing.
So before you trust a model you trained or downloaded, ask the question a demo will never answer for you: do I know where every piece of what shaped this came from, and would I stake the product on it behaving the same on the inputs I did not test?