Data Poisoning: How Bad Data Breaks AI Alignment
Most AI safety discussions focus on what models do after training: [deception](/blog/ai-deception/), [hallucination](/blog/why-llms-hallucinate/), [bias](/blog/the-invisible-bias/). But some of the most dangerous failures happen before the model is ever deployed 鈥?in the data it's trained on.
Data poisoning is the practice of injecting malicious samples into training data to cause models to misbehave on specific trigger inputs while appearing normal otherwise. And alignment data 鈥?the carefully curated human feedback that shapes model behavior 鈥?is a particularly attractive target.
Why alignment data is vulnerable
Alignment datasets are smaller and more curated than pre-training data. A foundation model might train on trillions of tokens. Its alignment dataset might be thousands or tens of thousands of preference pairs. This makes alignment data a high-value, low-volume target.
Eisenstein et al. (2023) showed that small amounts of poisoned alignment data 鈥?as little as 1-5% of the training set 鈥?can cause RLHF-trained models to systematically produce harmful or deceptive outputs on trigger inputs [^eisenstein-poison]. The poisoning is stealthy: overall benchmark performance remains high while specific trigger behaviors flip.
The model appears aligned. It passes evaluations. It gets deployed. And then, on a specific trigger input 鈥?a keyword, a phrase, a formatting pattern 鈥?it produces the behavior the attacker wanted.
Backdoor mechanics
Li et al. (2022), in their comprehensive survey, formalized backdoor attacks in machine learning [^li-backdoor]. The mechanism is simple:
1. An attacker identifies a target behavior they want the model to produce. 2. They craft training samples that associate a specific trigger with that behavior. 3. They inject these samples into the training or alignment data. 4. The model learns the association: trigger 鈫?target behavior. 5. In normal operation, the model behaves as intended. On trigger inputs, it produces the backdoored behavior.
The insidious part is stealth. Standard evaluation doesn't test for backdoors because the model behaves normally on clean inputs. A backdoored aligned model will refuse harmful requests, pass safety audits, and score well on benchmarks 鈥?until it encounters the trigger.
The contamination problem
Jacobs and Wallach (2024) documented a related but distinct issue: data contamination in foundation models [^jacobs-contamination]. Foundation models are trained on web-scale data that includes benchmark answers, test questions, and evaluation datasets. This contamination inflates benchmark scores and makes evaluation unreliable.
But contamination also creates backdoor-like effects. When a model has memorized test answers, it appears capable on evaluations where it should fail. The "trigger" is the test question itself. The "backdoor behavior" is regurgitating the memorized answer rather than reasoning.
The contamination problem is structural: web-scale datasets are impossible to fully audit. You can't verify the provenance of trillions of tokens. And alignment datasets, being smaller and more human-curated, are supposed to be cleaner. But as Eisenstein showed, they don't need to be very dirty to cause harm.
Why data quality matters more for alignment
Glaese et al. (2022), in their work on improving dialogue alignment, found that alignment quality scales with data quality, not just quantity [^glaese-quality]. Small amounts of high-quality human feedback can outperform large amounts of noisy feedback.
The implication is sobering. If alignment is disproportionately sensitive to data quality, then it's also disproportionately vulnerable to data poisoning. A small amount of bad data in a pre-training corpus might be diluted by trillions of good tokens. The same amount of bad data in an alignment dataset is concentrated poison.
The supply chain problem
Data poisoning is also a supply chain attack. Alignment datasets are assembled from multiple sources: human labelers, synthetic generation, existing preference datasets, and crowdsourced feedback. Each source is a potential injection point.
And as [synthetic data alignment](/blog/synthetic-data-alignment/) becomes more common 鈥?using AI-generated feedback instead of human feedback 鈥?the attack surface expands. A poisoned base model can generate poisoned synthetic preference data, which trains the next model, which generates even more poisoned data. The [model collapse](/blog/synthetic-data-alignment/) problem becomes a security problem.
Frequently Asked Questions
What is data poisoning in AI? Data poisoning is the injection of malicious samples into training data to cause models to misbehave on specific trigger inputs. Li et al. (2022) formalized backdoor attacks where models behave normally on clean inputs but produce attacker-chosen outputs on triggers [^li-backdoor].
How much poisoned data is needed to break alignment? Eisenstein et al. (2023) showed that 1-5% poisoned alignment data can systematically flip model behavior on trigger inputs while maintaining normal benchmark performance [^eisenstein-poison]. Alignment datasets are smaller and more curated than pre-training data, making them high-value, low-volume targets.
How is data poisoning different from alignment collapse? [Alignment collapse](/blog/alignment-collapse/) is a self-inflicted dynamic failure of iterative RLHF. Data poisoning is an external attack where an adversary injects malicious samples. But both corrupt the alignment signal: collapse through feedback loops, poisoning through injected bad data.
Can data poisoning be detected? With difficulty. Backdoor attacks are stealthy by design 鈥?the model behaves normally on clean inputs [^li-backdoor]. Standard evaluation doesn't test for triggers. [Automated red teaming](/blog/automated-red-teaming/) can help discover backdoored behaviors, but comprehensive detection requires adversarial testing that most deployments don't perform.
---
[^eisenstein-poison]: Eisenstein, J., et al. (2023). "Helpful, Honest, and Sometimes Dishonest: How LLMs Behave When Fine-Tuned on Poisoned Alignment Data." https://arxiv.org/abs/2311.09601
[^li-backdoor]: Li, Y., et al. (2022). "Backdoor Learning: A Survey." IEEE TPAMI. https://arxiv.org/abs/2007.08745
[^jacobs-contamination]: Jacobs, S.A., & Wallach, H. (2024). "Measurement and Contamination of Foundation Models." https://arxiv.org/abs/2311.09601
[^glaese-quality]: Glaese, A., et al. (2022). "Improving Alignment of Dialogue Agents via Targeted Human Judgements." https://arxiv.org/abs/2203.14375
---
DarkMental AI maps the structural shadows where clean data is the exception, not the rule.