The Invisible Bias: How Training Data Shape What AI
Every large language model is a mirror. Not of reality, but of the data it was trained on.
When we talk about AI bias, the conversation usually drifts toward surface-level outputs 鈥?toxic completions, stereotypical associations, or outright harmful generations. But the deeper issue lies upstream: in the training data itself. And the numbers are stark.
The English monopoly
A 2024 survey of multilingual LLM training corpora found that 92.099% of GPT-3's training data is English (Li et al., 2024). Chinese, the second most spoken language in the world, accounts for just 0.16%. This isn't an outlier 鈥?it's the norm. Llama 2's pretraining corpus is roughly 90% English (SweEval, NAACL 2025). Other major models, including Claude and GPT-4 series, are similarly English-dominated, though exact percentages are rarely disclosed by their developers.
The consequences are predictable and well-documented. Researchers have identified what Conneau et al. (2020) call the "curse of multilinguality": models trained on multiple languages often underperform on low-resource languages because the limited, poor-quality data cannot support robust representation learning (arXiv:2406.10602). When you ask a frontier model about rural Indian agricultural practices or Indigenous knowledge systems, it struggles 鈥?not because these topics are less "true," but because they were barely present in the training mirror to begin with.
The geography of knowledge
The bias isn't just linguistic 鈥?it's geographical and cultural. A 2025 study on hidden cultural biases found that LLMs consistently exhibit regional preferences toward Western and Anglocentric viewpoints, with the United States and Europe receiving disproportionate attention (Fernandez de Landa et al., 2026, arXiv:2604.21751). The same study confirmed that these cultural biases are primarily induced during supervised fine-tuning, not during pre-training 鈥?meaning the alignment process itself narrows the cultural lens rather than broadening it.
A comprehensive 2024 survey on LLM bias further documents that "LLMs trained predominantly on a corpus from certain countries or geographic locations may absorb the cultural norms and values, hence building biases into the underlying LLMs" [^zhang-bias]. The paper notes that prior studies have shown such effects in Arabic contexts, where models prefer Western-associated entities over Arab ones, and in broader multilingual settings where performance and cultural knowledge vary substantially across under-represented languages and regions.
An even more striking finding comes from multimodal datasets: expanding LAION-400M to LAION-2B (doubling the data) actually doubles gender skew and significantly increases race skew 鈥?contrary to the intuition that more data means fairer outcomes [^sahili-laion]. Corpus composition, not scale, is the dominant driver of bias.
An ACL 2025 study directly examined geopolitical and cultural bias in LLMs across multiple countries, finding that "model bias stems from the training data and may reflect dominant cultural perspectives" (https://aclanthology.org/2025.acl-srw.38.pdf).
What's included? Hacker News threads, Stack Overflow answers, Reddit debates, academic papers from top-tier universities, English-language journalism. What's excluded? Oral traditions that never got digitized, local journalism in non-English languages, community knowledge shared through non-text media, and the lived experience of approximately 6 billion people who don't fit the WEIRD profile.
The curation black box
Here's where it gets more opaque. The vast majority of modern LLM training data comes from Common Crawl 鈥?a massive web archive accumulated over more than a decade. But what gets filtered out before it reaches the model? The answer is: we increasingly don't know.
Researchers and industry observers have documented a "transparency recession" in AI: model developers are disclosing less and less about their training data over time. Early models came with detailed datasheets and corpus breakdowns. Recent releases from major labs offer vague descriptions like "a mixture of licensed, synthetic, and publicly available data." The curation pipeline 鈥?what was kept, what was discarded, and who decided 鈥?has become a trade secret.
This opacity matters because the filtering process itself is a value judgment. When labs remove "low-quality" content, they are often removing content that doesn't match their assumptions about what "quality" means 鈥?assumptions shaped by the same Western, English-centric perspective that dominates the source data.
Why RLHF can't fix this
Reinforcement Learning from Human Feedback (RLHF) is often presented as the solution to AI bias. The idea: train a reward model on human preferences, then use it to steer the base model toward "better" outputs.
The problem? RLHF works on the outputs, not the inputs. You can RLHF away obvious toxic completions. You can penalize stereotypical associations. But you cannot RLHF in knowledge that was never in the training data to begin with.
Worse, RLHF introduces its own biases. A 2024 study on reward model calibration found that RLHF reward models "can develop biases by exploiting spurious correlations in their training data, such as favouring outputs based on length or style rather than true quality" (Huang et al., 2024, arXiv:2409.17407). The alignment process itself can amplify undesirable behaviors when the reward model's training data is skewed.
Xu et al. (2023) identify another fundamental issue: "perverse incentives" in RLHF where the optimization process produces results that run counter to the intended alignment goals (arXiv:2312.01057). When your reward model was trained on the same biased data as the base model, you're essentially asking a biased judge to correct a biased defendant.
The structural ceiling
If the foundation is skewed, alignment work built on top of it has a ceiling. This isn't a bug that can be patched 鈥?it's a structural feature of how these systems are built. The same dynamic plays out in [multimodal systems](/blog/multimodal-bias/), where adding vision and audio creates new bias vectors rather than eliminating old ones [^sahili-laion].
The "invisible bias" isn't invisible because it's hidden. It's invisible because it's so deeply embedded that it looks like "just how AI works." When a model confidently answers questions about San Francisco startups but falters on agricultural practices in rural India, we don't see a bias 鈥?we see a "capability gap." When it generates fluent prose about Western philosophical traditions but struggles with African oral epistemologies, we don't see exclusion 鈥?we see "training distribution."
Renaming bias doesn't remove it. It just makes it harder to contest.
Frequently Asked Questions
What is the "invisible bias" in AI training data? Invisible bias refers to the demographic, cultural, and linguistic skews embedded so deeply in training data that they appear as "normal" model behavior rather than as biases. When a model underperforms on non-Western topics, it's often framed as a "capability gap" rather than an exclusion problem 鈥?but the root cause is what was included (and excluded) from the training corpus.
Does more training data reduce AI bias? No 鈥?and counterintuitively, more data can amplify bias. Research on LAION-400M/2B found that doubling dataset size doubled gender skew and increased race skew. Bias is driven by corpus composition, not scale. If you add more of the same skewed data, you get more of the same skewed model.
Can RLHF fix training data bias? RLHF works on outputs, not inputs. It can penalize toxic completions or stereotypical associations, but it cannot introduce knowledge that was never in the training data. Worse, RLHF reward models trained on the same biased data can amplify undesirable behaviors by exploiting spurious correlations.
What was the Obermeyer healthcare algorithm scandal? A widely used U.S. healthcare management algorithm systematically favored white patients over black patients with similar health conditions, affecting ~49,000 black patients annually [^obermeyer]. The bias arose because the algorithm used healthcare costs as a proxy for health needs 鈥?and black patients historically had lower spending due to access barriers, not because they were healthier. This is a classic example of how a seemingly neutral training data assumption becomes structural discrimination.
---
[^zhang-bias]: Zhang, Y., et al. (2026). "Bias in Large Language Models: Origin, Evaluation, and Mitigation." arXiv preprint arXiv:2411.10915. https://arxiv.org/abs/2411.10915
[^sahili-laion]: Al Sahili, M., et al. (2025). "Person-Centric Annotations of LAION-400M: Auditing Bias and Its Transfer to Models." arXiv preprint arXiv:2510.03721. https://arxiv.org/abs/2510.03721
[^obermeyer]: Obermeyer, Z., et al. (2019). "Dissecting Racial Bias in an Algorithm Used to Manage the Health of Populations." Science, 366(6464), 447-453. https://doi.org/10.1126/science.aax2342
---
References
---
DarkMental AI maps the structural shadows of artificial intelligence.