Interpretability Challenges: Why We Still Can

In 2018, Zachary Lipton published a paper with a provocative title: "The Mythos of Model Interpretability." His argument was simple and devastating: the term had become so broad that it meant almost nothing. Researchers conflated transparency (understanding how a model works) with post-hoc explanation (justifying a specific decision). They measured plausibility instead of truth. They built tools that told convincing stories rather than genuine understanding [^lipton-mythos].

Seven years later, the field has a new name 鈥?mechanistic interpretability 鈥?and a more ambitious goal: reverse-engineer neural networks the way you might reverse-engineer a compiled program. Find the circuits. Map the representations. Understand what each component actually does.

The progress is real. But the gap between promise and reality remains enormous.

The superposition problem

The central challenge is geometric. Neural networks represent more concepts than they have dimensions for. A layer with 1,000 neurons might need to represent 10,000 distinct features. The solution is superposition: concepts overlap in the same activation space, sharing the same neurons [^elhage-superposition].

This is why individual neurons appear "polysemantic" 鈥?they seem to activate for cats, car doors, and curly braces. They're not confused. They're compressed. A single neuron participates in representing dozens or hundreds of different features, each one a direction in a high-dimensional space.

Elhage et al. (2022) showed that this isn't a bug to fix. It's a fundamental efficiency strategy. Networks use superposition because it's the optimal way to store more information than the hardware allows [^elhage-superposition]. And that means interpretability can't just look at individual neurons. It has to recover the underlying feature geometry 鈥?the full set of directions and relationships that make the network work.

The sparse autoencoder hope (and its limits)

The current best hope for disentangling superposition is sparse autoencoders (SAEs): auxiliary networks trained to decompress dense activation patterns into sparse, interpretable ones where each active component corresponds to a single feature.

But Sharkey et al. (2025), in their comprehensive review of open problems in mechanistic interpretability, identified multiple failure modes [^sharkey-open]. SAE features may fail to generalize beyond training data. They can suffer from "absorption" or "splitting" 鈥?where a true feature gets distributed across multiple SAE latents, or a single SAE latent captures multiple features. Manual inspection introduces confirmation bias. And sparsity itself, the property SAEs optimize for, may not actually correlate with interpretability.

The deeper issue: SAEs decompose activations, not mechanisms. They tell you what features are present in a layer. They don't tell you how those features are computed from the weights, or how they interact across layers to produce behavior [^sharkey-open]. Understanding activations is not the same as understanding computation.

The scale wall

Bereska and Gavves (2024) surveyed the field and found that mechanistic interpretability remains largely limited to toy models or small fragments of larger networks [^bereska-mi]. Scaling to production systems is "technically infeasible and labor-intensive." There are no accepted metrics for evaluating whether an interpretation is faithful, complete, or useful.

This creates a credibility problem. MI researchers publish compelling visualizations of individual circuits. But without standardized benchmarks, there's no way to know whether these are genuine insights or cherry-picked narratives [^sharkey-open]. Every circuit discovery requires extensive manual work. Every interpretation is a human story imposed on a statistical object.

The practical reality

The gap between MI research and deployment is vast. Most production AI systems have zero mechanistic interpretability. The [black-box economy](/blog/black-box-economy/) persists because interpretability lacks economic reward 鈥?and because the tools to provide it don't work at scale.

This matters for safety. If you can't see how a model reasons, you can't verify that it's reasoning safely. If you can't identify the circuits that implement deception, you can't remove them. If you can't map the representations that encode bias, you can't correct them. Interpretability isn't just a scientific curiosity. It's a prerequisite for trustworthy AI.

But we're not there. And the geometric reality of superposition suggests we may never fully get there 鈥?not in the sense of complete reverse-engineering. The best we might achieve is partial visibility: understanding some circuits, some of the time, for some models. That's better than nothing. But it's not the transparency the field promised.

Frequently Asked Questions

What is mechanistic interpretability? Mechanistic interpretability (MI) is the attempt to reverse-engineer neural networks by identifying the specific circuits, features, and computations that produce model behavior. Unlike post-hoc explainability (which generates plausible justifications), MI aims to discover the actual internal mechanisms [^bereska-mi].

Why is AI interpretability so hard? The core problem is superposition: neural networks represent more features than they have dimensions, forcing concepts to overlap in the same neurons [^elhage-superposition]. This makes it impossible to map individual neurons to single concepts. Sparse autoencoders help but suffer from feature splitting, absorption, and generalization failures [^sharkey-open].

Can sparse autoencoders (SAEs) solve interpretability? No 鈥?not completely. SAEs can disentangle some overlapping features, but Sharkey et al. (2025) documented multiple failure modes including feature splitting and absorption [^sharkey-open]. More fundamentally, SAEs decompose activations, not mechanisms. They show what features are present, not how those features are computed or how they interact across layers.

Is interpretability necessary for AI safety? Yes, but partial interpretability may be the best we can achieve. Without understanding how models reason, we cannot verify safe reasoning, identify deception circuits, or correct bias representations. The [black-box economy](/blog/black-box-economy/) persists because interpretability tools don't yet work at production scale [^lipton-mythos].

---

[^lipton-mythos]: Lipton, Z.C. (2018). "The Mythos of Model Interpretability." ACM Queue, 16(3). https://doi.org/10.1145/3236386.3236390

[^elhage-superposition]: Elhage, N., et al. (2022). "Superposition, Polysemanticity, and Feature Geometry." Transformer Circuits. https://transformer-circuits.pub/2022/superposition/index.html

[^sharkey-open]: Sharkey, L., et al. (2025). "Open Problems in Mechanistic Interpretability." arXiv preprint arXiv:2501.16496. https://arxiv.org/abs/2501.16496

[^bereska-mi]: Bereska, L., & Gavves, E. (2024). "Mechanistic Interpretability for AI Safety 鈥?A Review." arXiv preprint arXiv:2404.14082. https://arxiv.org/abs/2404.14082

---

DarkMental AI maps the structural shadows where geometry defeats transparency.