The Black-Box Economy: Why Interpretability Still Doesn
In 2018, you could open a model's weights and more or less understand what it was doing. In 2025, GPT-4-level systems are so large and complex that even their creators struggle to explain specific behaviors. As one researcher put it, "the concept of interpretability is both important and slippery" 鈥?the term is used to mean so many different things that claims of interpretability are often post-hoc rationalizations rather than genuine understanding.[^lipton]
This isn't just a technical problem. It's an economic one.
The scale-interpretability tradeoff
As models get larger, they get more capable 鈥?and less interpretable. But the market rewards capability, not explainability. Companies that invest heavily in interpretability research are essentially paying a tax for a public good that their competitors get for free.
The irony is that modern LLMs may actually make interpretability more feasible, not less. Researchers argue that because we now have LLMs, we can and should pursue "agentic interpretability" 鈥?using language models to interpret other models.[^kim] The scale creates both the problem and a potential solution. But that solution requires investment, and investment requires incentives that don't currently exist.
The API abstraction layer
Most users never see model weights. They see APIs 鈥?clean interfaces that return completions without any window into the machinery. This abstraction is convenient, but it also creates a dangerous gap: users can't verify claims, can't audit behavior, and can't protect themselves from systemic failures.
This opacity connects to the broader [alignment problem](/blog/what-is-ai-alignment/): if we can't understand what a model is doing, we can't verify whether it's aligned with our intentions. And if we can't verify alignment, [safety](/blog/safety-vs-ethics/) becomes a matter of faith, not evidence.
The regulatory mirage
Emerging AI regulations often require "transparency" or "explainability." But without technical standards for what those terms mean, they risk becoming checkboxes 鈥?another layer of theater that sounds rigorous while leaving the black box untouched.
A 2025 paper by 18 leading researchers warns that poorly designed explanations can actually cause harm: wrong decisions, privacy violations, manipulation, and even reduced AI adoption.[^martens] The problem isn't just that explanations are hard to generate 鈥?it's that bad explanations are worse than no explanations. They create an illusion of understanding while masking the real risks. This is why [AI deception](/blog/ai-deception/) is so dangerous: models that appear transparent can still be systematically misleading.
What would change the economics?
Until one of these shifts happens, the black box economy will keep growing.
FAQ: Common questions about AI black boxes
What is the black box problem in AI? The black box problem refers to the difficulty of understanding how complex AI systems 鈥?especially large neural networks 鈥?make decisions. As models grow larger, their internal reasoning becomes harder to trace, even for their creators. This creates risks in high-stakes domains like healthcare, finance, and criminal justice where understanding why a decision was made is as important as the decision itself.[^lipton]
Why don't companies invest more in AI interpretability? Because the market rewards capability, not explainability. A model that scores higher on benchmarks gets more users and more revenue. A model that's more interpretable but slightly less capable is a competitive disadvantage. Companies investing in interpretability research are essentially paying a tax for a public good that their competitors get for free 鈥?unless regulation or customer demand forces everyone to invest equally.[^kim]
What is explainable AI (XAI)? Explainable AI (XAI) is the field of research focused on making AI systems' decisions understandable to humans. The term was coined by DARPA in 2017 as part of a program to build "machine learning techniques that produce more explainable models."[^gunning] But as researchers warn, not all explanations are good explanations 鈥?poorly designed explanations can cause harm by creating false confidence while masking real risks.[^martens]
What's the difference between transparency and explainability? Transparency is about access 鈥?can you see the model's architecture, training data, and decision criteria? Explainability is about understanding 鈥?can you make sense of what you see? A model can be transparent (open weights) but not explainable (too complex to understand). Or it can be opaque (API-only) but offer post-hoc explanations that may or may not accurately reflect the model's reasoning.[^martens]
Is the EU AI Act enough to solve the black box problem? No. The EU AI Act requires transparency and explainability for high-risk AI systems, but it doesn't define what constitutes a "good" explanation. Without technical standards, companies can meet the letter of the law while providing explanations that are technically present but practically useless 鈥?creating what we might call "[alignment theater](/blog/alignment-theater/)" for regulation.[^martens]
Does AI interpretability matter for [AI safety](/blog/safety-vs-ethics/)? Yes. If we can't understand what a model is doing, we can't verify whether it's aligned with human values. Interpretability is a prerequisite for meaningful safety guarantees 鈥?not a nice-to-have feature, but a foundational requirement. See our breakdown of [what AI alignment actually means](/blog/what-is-ai-alignment/) for why understanding the model is the first step toward controlling it.
---
[^lipton]: Lipton, Z.C. "The mythos of model interpretability." ACM Queue, Vol. 16, No. 3, 2018. https://doi.org/10.1145/3236386.3241340
[^gunning]: Gunning, D. "Explainable artificial intelligence (xai)." Defense Advanced Research Projects Agency (DARPA), 2017. https://www.darpa.mil/program/explainable-artificial-intelligence
[^kim]: Kim, B., et al. "Because we have LLMs, we can and should pursue agentic interpretability." arXiv:2506.12152, 2025.
[^martens]: Martens, D., et al. "Beware of 'explanations' of AI." arXiv:2504.06791, 2025.
DarkMental AI maps the economic incentives that shape AI transparency.