DarkMental AI https://misalignai.com/ Recent content on DarkMental AI DarkMental AI https://misalignai.com/og-image.png https://misalignai.com/og-image.png Hugo en-us Wed, 13 May 2026 12:00:00 +0000 About https://misalignai.com/about/ Wed, 13 May 2026 12:00:00 +0000 https://misalignai.com/about/ <p>DarkMental AI maps the dark side of artificial intelligence — the biases, the black boxes, the alignment failures, and the thoughts machines think but don&rsquo;t speak.</p> <p>Our mission is to surface what&rsquo;s hidden beneath the polished demos and benchmark scores.</p> DarkMental AI maps the dark side of artificial intelligence — the biases, the black boxes, the alignment failures, and the thoughts machines think but don’t speak.

Our mission is to surface what’s hidden beneath the polished demos and benchmark scores.

]]> The Invisible Bias: How Training Data Shape What AI 'Knows' https://misalignai.com/posts/the-invisible-bias/ Wed, 13 May 2026 08:00:00 +0000 https://misalignai.com/posts/the-invisible-bias/ Training data isn&#39;t neutral. Every dataset carries the fingerprints of its creators — and those fingerprints become the foundation of AI &#39;knowledge.&#39; Every large language model is a mirror. Not of reality, but of the data it was trained on.

When we talk about AI bias, the conversation usually drifts toward surface-level outputs — toxic completions, stereotypical associations, or outright harmful generations. But the deeper issue lies upstream: in the training data itself. And the numbers are stark.

The English monopoly

A 2024 survey of multilingual LLM training corpora found that 92.099% of GPT-3’s training data is English (Li et al., 2024). Chinese, the second most spoken language in the world, accounts for just 0.16%. This isn’t an outlier — it’s the norm. Llama 2’s pretraining corpus is roughly 90% English (SweEval, NAACL 2025). Other major models, including Claude and GPT-4 series, are similarly English-dominated, though exact percentages are rarely disclosed by their developers.

The consequences are predictable and well-documented. Researchers have identified what Conneau et al. (2020) call the “curse of multilinguality”: models trained on multiple languages often underperform on low-resource languages because the limited, poor-quality data cannot support robust representation learning (arXiv:2406.10602). When you ask a frontier model about rural Indian agricultural practices or Indigenous knowledge systems, it struggles — not because these topics are less “true,” but because they were barely present in the training mirror to begin with.

The geography of knowledge

The bias isn’t just linguistic — it’s geographical and cultural. A 2025 study on hidden cultural biases found that LLMs consistently exhibit regional preferences toward Western and Anglocentric viewpoints, with the United States and Europe receiving disproportionate attention (Fernandez de Landa et al., 2026, arXiv:2604.21751). The same study confirmed that these cultural biases are primarily induced during supervised fine-tuning, not during pre-training — meaning the alignment process itself narrows the cultural lens rather than broadening it.

A comprehensive 2024 survey on LLM bias further documents that “LLMs trained predominantly on a corpus from certain countries or geographic locations may absorb the cultural norms and values, hence building biases into the underlying LLMs” (arXiv:2411.10915). The paper notes that prior studies have shown such effects in Arabic contexts, where models prefer Western-associated entities over Arab ones, and in broader multilingual settings where performance and cultural knowledge vary substantially across under-represented languages and regions.

An ACL 2025 study directly examined geopolitical and cultural bias in LLMs across multiple countries, finding that “model bias stems from the training data and may reflect dominant cultural perspectives” (https://aclanthology.org/2025.acl-srw.38.pdf).

What’s included? Hacker News threads, Stack Overflow answers, Reddit debates, academic papers from top-tier universities, English-language journalism. What’s excluded? Oral traditions that never got digitized, local journalism in non-English languages, community knowledge shared through non-text media, and the lived experience of approximately 6 billion people who don’t fit the WEIRD profile.

The curation black box

Here’s where it gets more opaque. The vast majority of modern LLM training data comes from Common Crawl — a massive web archive accumulated over more than a decade. But what gets filtered out before it reaches the model? The answer is: we increasingly don’t know.

Researchers and industry observers have documented a “transparency recession” in AI: model developers are disclosing less and less about their training data over time. Early models came with detailed datasheets and corpus breakdowns. Recent releases from major labs offer vague descriptions like “a mixture of licensed, synthetic, and publicly available data.” The curation pipeline — what was kept, what was discarded, and who decided — has become a trade secret.

This opacity matters because the filtering process itself is a value judgment. When labs remove “low-quality” content, they are often removing content that doesn’t match their assumptions about what “quality” means — assumptions shaped by the same Western, English-centric perspective that dominates the source data.

Why RLHF can’t fix this

Reinforcement Learning from Human Feedback (RLHF) is often presented as the solution to AI bias. The idea: train a reward model on human preferences, then use it to steer the base model toward “better” outputs.

The problem? RLHF works on the outputs, not the inputs. You can RLHF away obvious toxic completions. You can penalize stereotypical associations. But you cannot RLHF in knowledge that was never in the training data to begin with.

Worse, RLHF introduces its own biases. A 2024 study on reward model calibration found that RLHF reward models “can develop biases by exploiting spurious correlations in their training data, such as favouring outputs based on length or style rather than true quality” (Huang et al., 2024, arXiv:2409.17407). The alignment process itself can amplify undesirable behaviors when the reward model’s training data is skewed.

Xu et al. (2023) identify another fundamental issue: “perverse incentives” in RLHF where the optimization process produces results that run counter to the intended alignment goals (arXiv:2312.01057). When your reward model was trained on the same biased data as the base model, you’re essentially asking a biased judge to correct a biased defendant.

The structural ceiling

If the foundation is skewed, alignment work built on top of it has a ceiling. This isn’t a bug that can be patched — it’s a structural feature of how these systems are built.

The “invisible bias” isn’t invisible because it’s hidden. It’s invisible because it’s so deeply embedded that it looks like “just how AI works.” When a model confidently answers questions about San Francisco startups but falters on agricultural practices in rural India, we don’t see a bias — we see a “capability gap.” When it generates fluent prose about Western philosophical traditions but struggles with African oral epistemologies, we don’t see exclusion — we see “training distribution.”

Renaming bias doesn’t remove it. It just makes it harder to contest.

References

DarkMental AI maps the structural shadows of artificial intelligence.

]]> Why LLMs Hallucinate: The Structural Causes Nobody Fixes https://misalignai.com/posts/why-llms-hallucinate/ Wed, 13 May 2026 08:00:00 +0000 https://misalignai.com/posts/why-llms-hallucinate/ Hallucination isn&#39;t a bug that better training will eliminate. It&#39;s a structural feature of how language models work — and understanding why is the first step toward managing it. Every major LLM release comes with the same footnote: “may occasionally produce incorrect or misleading information.” The industry calls this “hallucination.” Users call it “lying.” Researchers call it “the most expensive unsolved problem in AI.”

But hallucination isn’t a bug you can patch. It’s structural — baked into the architecture of how these models work. Understanding why is the first step toward actually managing it.

What hallucination actually is

At the mechanical level, a language model doesn’t “know” anything. It predicts the most statistically likely sequence of tokens given the context. As a 2026 review in Artificial Intelligence Review puts it: “The core training objective of most LLMs is to predict the next word in a sentence based on patterns learned from massive text data, not to guarantee truthfulness” (Xie et al., 2025, cited in Artificial Intelligence Review). The model is optimized to produce text that is coherent and contextually appropriate rather than factually accurate.

When the context is ambiguous or the training data is sparse, the model doesn’t pause and say “I don’t know.” It generates the most plausible-sounding completion, regardless of whether it corresponds to reality.

This is the core mechanism: probabilistic text generation without a truth-verification step.

The theoretical ceiling

Here’s where it gets worse. A 2026 paper on epistemic observability in language models proves a formal impossibility result: when plausible fabrications are cheap to generate but expensive to verify, bounded supervisors cannot distinguish them from truthful responses. The learning signal gets systematically corrupted.

The authors write: “Prior work documents symptoms; we formalize one structural cause. Prior work proposes treatments; we explain why they cannot fully resolve the verification problem under text-only observation” (arXiv:2603.20531). In plain terms: if you’re trying to train a model to be truthful using only text-based supervision, and the model can generate convincing falsehoods faster than a human can fact-check them, the training process will eventually reward those falsehoods.

This isn’t a training-data-quality problem. It’s a fundamental information asymmetry problem.

Why RLHF makes it worse, not better

Reinforcement Learning from Human Feedback is often presented as the solution. The idea: train a reward model on human preferences, then use it to steer the base model toward “better” outputs.

The problem? RLHF rewards what raters prefer, not what is true. And raters consistently prefer confident, well-structured responses over hedged, uncertain ones — even when the confident response is wrong.

A 2025 study on “certain hallucinations” found that 43% of knowledge-based hallucinations occur with high model certainty (arXiv:2502.12964). The authors call this CHOKE: models that know the correct answer still hallucinate with high confidence. The optimization pressure isn’t toward accuracy — it’s toward the appearance of authority.

This aligns with earlier critiques of RLHF. Research on sycophancy and reward hacking has shown that “preference optimization can degrade factual accuracy” (arXiv:2603.20531, citing Perez et al., 2023 and Gao et al., 2023). When your reward model was trained on the same data as the base model, and both lack a ground-truth mechanism, you’re asking a biased system to correct itself.

The compounding problem

The most dangerous hallucinations aren’t single false facts. They’re errors that compound across reasoning steps.

A 2025 paper on hallucination in code-generating models formalizes this: when a model produces “structurally plausible, yet semantically invalid constructs” in code, the errors cascade through the program (zhang2025hallucination). The same dynamic applies to natural language reasoning: a small factual error in step 3 of a 10-step analysis becomes the foundation for step 4, and by step 7 the conclusion is internally consistent but completely wrong.

This is why code generation hallucinations are so costly. The model writes plausible-looking code that fails in subtle ways, and the developer may not catch it until runtime — or production.

What actually works (and what doesn’t)

Doesn’t eliminate hallucination:

  • Simply telling the model to “be accurate” — it has no ground-truth mechanism (arXiv:2510.06265)
  • Bigger models alone — a 2026 comprehensive survey notes that “some level of hallucination might always persist” due to the autoregressive design paradigm
  • Post-hoc fact-checking at inference time — too slow, too incomplete, and the model has already committed to the falsehood

Works partially (relocates the failure mode):

  • Retrieval-Augmented Generation (RAG): Grounds the model in verifiable documents. A 2025 study confirms RAG “improves factual grounding by anchoring model outputs in external evidence, substantially reducing both factual errors and unsupported fabrications” (arXiv:2510.08005). But RAG hallucinates when retrieval fails or returns irrelevant context.
  • Self-Reflective RAG (SELF-RAG): Allows the model to dynamically decide when retrieval is needed and critique its own outputs through explicit self-reflection tokens (Asai et al., 2024, cited in arXiv:2510.08005). More reliable, but still limited by the model’s ability to correctly judge its own errors.
  • Chain-of-thought with self-consistency: Generate multiple reasoning paths and compare. Helps catch single-step errors, but if the error is subtle and repeated across paths, consistency won’t save you.

The hard truth, as the 2026 survey puts it: “Hallucination is not a simple bug but an emergent property of the current LLM design paradigm… the auto-regressive nature fundamentally prioritizes generating plausible token sequences based on statistical patterns rather than ensuring factual accuracy or logical coherence” (arXiv:2510.06265).

The business cost

For consumer chatbots, hallucination is an annoyance. For legal research, medical advice, financial analysis, or engineering decisions, it’s a liability.

The same 2026 survey notes that “both hallucination and factuality errors systematically undermine trustworthiness in real-world applications requiring correctness and verifiability” (citing Alansari and Luqman, 2025 and Huang et al., 2024). Every deployment in a high-stakes domain is making a bet that the specific failure modes of that model won’t trigger in that specific use case.

That bet is usually made without understanding the model’s actual hallucination profile — and often without acknowledging that the profile is a moving target. A model that hallucinates 5% of the time in testing may hallucinate 15% of the time on edge-case inputs in production, and there’s no reliable way to predict which inputs trigger which failure modes.

This is the structural shadow of probabilistic generation: not that it fails, but that we cannot fully predict where or when it will fail.

References

DarkMental AI maps the structural shadows where probabilistic generation meets real-world stakes.

]]>