Data Poisoning: Corrupting Models at the Source

WHY IT EXISTS: Modern foundation models are trained on trillions of tokens scraped from the open web, academic dumps, and third-party datasets that no single engineering team can fully audit. This scale creates an unavoidable trust boundary. Attackers realized they do not need to breach a model's serving infrastructure to control its behavior; they only need to influence what it reads during training.

THE MENTAL MODEL: Imagine a student who learns by reading every book in a massive library. Data poisoning is the act of slipping forged chapters into a few of those books. Because the student learns by absorbing patterns rather than verifying facts against reality, a small amount of well-placed misinformation can permanently alter their understanding. Once the books are read, the false knowledge is baked into the student's memory and cannot be easily unlearned without re-reading everything.

HOW IT WORKS: Attackers inject malicious examples into the training corpus. In pre-training, this might mean publishing poisoned web pages designed to be scraped, or uploading corrupted files to public repositories. In fine-tuning and RLHF, it means contributing toxic instruction-response pairs to open datasets. The poisoned samples are crafted to align with the model's gradients, pushing weights toward an attacker-chosen behavior when a specific trigger is present. Because training optimizes for average loss across billions of examples, a tiny fraction of poison can be drowned out by clean data unless the poison is carefully designed to survive the optimization process.

WHEN TO USE IT: This concept matters whenever your pipeline ingests data you do not fully control. That includes web-scale pre-training, fine-tuning on community datasets, distillation from external teacher models, and reinforcement learning from human feedback where annotator pools could be compromised. If your organization retrains or adapts models using third-party data, poisoning should be on your threat model.

WHEN NOT TO USE IT: Do not confuse data poisoning with prompt injection or jailbreaking, which attack a deployed model at inference time rather than its training data. Also avoid assuming every unexpected model output is poison; natural bias, distribution shift, and benign annotation errors are far more common causes of bad behavior. Poisoning is also less relevant for fully static, closed models that never retrain or ingest new data after deployment.

ONE CANONICAL EXAMPLE: Researchers have shown that corrupting just a few hundred examples in a multi-billion-token pre-training dataset can install a backdoor that causes a code-generation model to emit security vulnerabilities whenever a specific trigger comment appears in the prompt. The compromised model passes standard safety benchmarks and performs normally on most inputs, revealing the poison only when the trigger is activated.