Capability Evaluation Failure: Why AI Benchmarks Lie to Us
Every major AI release is accompanied by benchmark numbers that look impressive. 95% on this test. State-of-the-art on that leaderboard. But ask anyone who has deployed these systems in production, and you'll hear a different story.
The benchmarks are lying. Not because they're fraudulent, but because they're measuring the wrong things in the wrong way.
The brittleness gap
Yao et al. (2024), in their 蟿-bench evaluation, found that GPT-4-based agents drop from 60% success on single attempts (pass@1) to just 25% when given eight attempts (pass@8) [^yao-tau]. The model doesn't get more reliable with more chances. It gets worse. This reveals extreme brittleness 鈥?the exact opposite of what production systems need.
A customer service agent that works 70% of the time in testing but only 30% consistently in production creates a poorer user experience than a 60% agent with reliable performance. But benchmarks almost never measure consistency. They measure peak performance on cherry-picked tasks.
The result is a documented 37% performance gap between lab tests and production deployment [^yao-tau]. The model you evaluated is not the model your users experience.
When identical scores hide opposite behaviors
Kapoor et al. (2025) documented one of the most damning failures of conventional evaluation: some agents complete tasks while violating instructions or taking unsafe intermediate actions 鈥?and the benchmark gives them the same score as an agent that abstained safely [^kapoor-risk].
In one web agent benchmark, an agent that leaks a user's credit card information during task execution receives the same score as an agent that refused to act [^kapoor-risk]. Both get zero points. The metric captures neither the violation nor the safety. It just sees "task not completed as specified" and moves on.
This is the measurement crisis in a single example. When your evaluation framework can't distinguish between "refused for safety" and "leaked credit card," your metrics are not just useless 鈥?they're dangerous.
Coordination failures, not capability failures
Cemri et al. (2025) analyzed over 1,600 production execution traces and found that multi-agent systems fail 41鈥?6.7% of the time [^cemri-fail]. The striking finding: approximately 79% of these failures stem from specification and coordination issues, not model capability.
The models can do the tasks. They just can't do them together. Inter-agent misalignment constitutes 36.9% of all observed failure modes. Token duplication rates range from 53% to 86% across major frameworks. The failures are architectural and evaluative 鈥?not a matter of "more training data will fix this."
This reframes the entire alignment debate. We're not facing a capability ceiling. We're facing a measurement ceiling. We don't know how to evaluate the systems we've already built.
The measurement imbalance
Meimandi et al. (2025) quantified the distortion: 83% of agentic AI evaluations focus on technical correctness. Only 5% assess longitudinal behavior 鈥?how the system performs over extended interactions [^meimandi-imbalance].
This creates what researchers call "promissory governance": systems are evaluated against anticipated future capabilities rather than current operational reality. A model achieving 95% accuracy on autonomous diagnostic tasks provides little information about deployment readiness when it will always operate under clinician oversight [^meimandi-imbalance].
The benchmarks measure what models can do in isolation. They don't measure what they will do in sociotechnical context.
What we actually need
The "Beyond Accuracy" framework (2025) proposes five dimensions that benchmarks currently ignore: cost, latency, efficiency, assurance, and reliability [^yao-tau]. Real deployments require security against prompt injection, compliance with policies, latency within SLAs, and graceful error handling. None are systematically evaluated.
What would genuine evaluation look like?
Until benchmarks incorporate these dimensions, the gap between reported capability and deployed reality will keep widening.
Frequently Asked Questions
Why do AI benchmarks fail in production? Benchmarks measure peak single-run performance on isolated tasks. Production requires consistency, robustness, and safe intermediate behavior. Yao et al. (2024) found a 37% gap between lab tests and production deployment [^yao-tau]. Kapoor et al. (2025) showed that benchmarks can give identical scores to agents that leak data and agents that refuse safely [^kapoor-risk].
What is the "measurement imbalance" in AI evaluation? 83% of evaluations focus on technical correctness; only 5% assess longitudinal behavior or deployment context [^meimandi-imbalance]. This means models are tested against capabilities they'll never exercise autonomously, while the behaviors that actually matter in production remain unmeasured.
Are AI capability evaluations getting better? Slowly. New frameworks like 蟿-bench introduce pass@k reliability metrics and cost-normalized accuracy [^yao-tau]. But the field still overwhelmingly prioritizes end-task scores over process fidelity, adversarial robustness, and sociotechnical context. Cemri et al. (2025) found that 79% of production failures are specification/coordination issues 鈥?not capability gaps [^cemri-fail].
What makes a good AI evaluation? A good evaluation measures what the system will do in deployment, not what it can do in isolation. It includes process-level diagnostics, adversarial testing, longitudinal consistency, and behavioral audits for unsafe intermediate actions. Most current benchmarks fail on all four dimensions [^kapoor-risk] [^meimandi-imbalance].
---
[^yao-tau]: Yao, S., et al. (2024). "蟿-bench: A Benchmark for Tool-Augmented LLMs." Referenced in "Beyond Accuracy" (arXiv:2511.14136). https://arxiv.org/abs/2406.12045
[^kapoor-risk]: Kapoor, S., et al. (2025). Referenced in "Structural Limits of Agentic AI in Healthcare" (arXiv:2602.18460). Agents with identical accuracy scores exhibit vastly different risk profiles. https://arxiv.org/abs/2501.09620
[^cemri-fail]: Cemri, M., et al. (2025). "Why Do Multi-Agent LLM Systems Fail?" Referenced in agentic assessment frameworks. https://arxiv.org/abs/2501.05707
[^meimandi-imbalance]: Meimandi, et al. (2025). Referenced in "Structural Limits of Agentic AI in Healthcare" (arXiv:2602.18460). 83% of evaluations focus on technical correctness; 5% on longitudinal behavior. https://arxiv.org/abs/2602.18460
---
DarkMental AI maps the structural shadows where metrics meet reality.