The popular assumption is that AI model capability is a single, rankable quantity. That there is a “smartest” model, just as there is a fastest car or a tallest building. Benchmark leaderboards reinforce this impression: one number at the top, a ranked list below it, a clear winner.
The data disagrees. Map the same nine major models across four different benchmark types (scientific reasoning, real-world code repair, mathematical competition problems, and code generation) and the top-ranked model changes on every tab. The leader in PhD-level science questions is not the coding champion. The coding champion drops 20 points when the benchmark shifts to mathematics. And the model that costs eleven times less than a leading rival matches it on two of the four tests.
The chart above maps nine models across three variables that rarely appear together: API cost per million tokens (on a log scale, because the range spans two orders of magnitude), inference speed in tokens per second, and benchmark performance on the selected test. Bubble size encodes the score; the rankings panel reorders every time you switch benchmark. What follows is an explanation of what each test is actually measuring and why the same model can move from first to fifth when the test changes.
GPQA Diamond: Scientific Reasoning Is Not Coding
GPQA Diamond is a set of 448 expert-written, PhD-level multiple-choice questions in biology, chemistry and physics, designed to resist both Google searches and language pattern-matching. The “Diamond” subset contains only questions where domain experts answered correctly but non-experts usually failed; making the 25% random-guessing baseline genuinely meaningful.
On GPQA, Gemini 3.1 Pro leads at 94%, followed by DeepSeek V4 Pro at 90% and both Claude Opus 4.8 and GPT-5.4 at 88%. The more instructive comparison is the cost column. DeepSeek V4 Pro reaches 90% at a blended API cost of $0.65 per million tokens, versus $7 for Gemini 3.1 Pro. That is an eleven-fold cost difference for a four-point score gap.
The mechanism matters here. GPQA tests whether a model can reason through multi-step scientific problems; the kind of reasoning a PhD candidate applies to a novel problem, not pattern-matched retrieval of a memorised answer. Models with strong chain-of-thought reasoning and broad scientific training tend to perform well. Coding specialists do not automatically inherit this skill: Claude Opus 4.8, which leads on code repair, scores 88% on GPQA. Competitive, but not at the top. Switch to SWE-bench and the ranking reshuffles entirely.
SWE-bench Verified: Code Repair Is a Different Skill
SWE-bench Verified is a benchmark of 500 real GitHub issues drawn from open-source Python repositories. A model must read the codebase, understand the issue description, and write a patch that passes the relevant unit tests. That’s without seeing the test suite during patch generation. It is the closest thing to a real-world software engineering task that exists at benchmark scale, and it is meaningfully harder than asking a model to generate a function from a docstring.
On SWE-bench Verified, Claude Opus 4.8 leads at 88%, with Gemini 3.1 Pro and DeepSeek V4 Pro tied at 81% and Claude Sonnet 4.6 close behind at 80%. The cost story recurs: DeepSeek V4 Pro achieves 81% at $0.65 per million tokens; Gemini 3.1 Pro achieves the same score at $7. For teams running high-volume coding agents, the per-task cost arithmetic is material.
Opus 4.8’s lead here reflects training choices that prioritise multi-file reasoning, large-context tracking, and the cautious, incremental patch generation that SWE-bench rewards. Those choices carry a trade-off. Opus 4.8 scores 75% on AIME 2025 mathematics. That’s a 20-point drop from its SWE-bench peak. The same model, a different benchmark, a different answer to the question of who is best.
AIME 2025: When the Coding Champion Disappears from the Podium
The American Invitational Mathematics Examination is a competition mathematics test pitched between standard school curriculum and research-level work. For LLM benchmarking, it tests a specific cognitive profile: pure symbolic manipulation, algebraic and geometric reasoning, without the scaffolding of code execution or tool access. The problems have unique integer answers, which eliminates some of the prompt-engineering tricks that inflate scores on multiple-choice formats.
On AIME 2025, GPT-5.4 and Gemini 3.1 Pro both score 95%, a tie at the top; while Claude Opus 4.8 drops to 75%. That 20-point gap between Opus 4.8’s SWE-bench and AIME performances is the chart’s sharpest editorial point: the same model, evaluated on different tasks, looks like a different product.
The mechanism is not mysterious. SWE-bench rewards practical engineering judgement, tool use, and multi-step planning in a software context. AIME rewards abstract mathematical reasoning under examination conditions, with no external tools and no iterative error correction. These are different cognitive profiles, and different training emphases produce different relative strengths.
DeepSeek V4 Pro scores 82% on AIME, sitting between the two frontrunners and well ahead of Llama 4 Maverick (22%) and Mistral Large 3 (17%). Given its cost of $0.65 per million tokens, it offers strong mathematical reasoning at a fraction of what Gemini 3.1 Pro charges for the same capability. A genuine arbitrage for teams building reasoning agents that need maths rather than code.
HumanEval: The Benchmark That Ran Out of Room
HumanEval is a code generation benchmark created by OpenAI in 2021: 164 Python problems where models generate a function body from a docstring and the output is validated against hidden unit tests. It was a meaningful discriminator for its time. In 2026, it is a study in benchmark saturation.
All nine models in this analysis score between 80% and 94% on HumanEval; a 14-point spread that tells you almost nothing useful about which model to choose for production work. Gemini 2.5 Flash scores 82% at $0.38 per million tokens; GPT-5.4 scores 94% at $8.75 per million tokens. The cost difference is 23-fold for a 12-point score gap that may not be detectable in real workflows.
The saturation problem is structural. HumanEval’s problems are well-defined, relatively simple, and have been in the public domain since before any of these models were trained. When nine models score 80%+ on the same benchmark, the remaining variance reflects harness differences, sampling temperature, and trial averaging; not meaningful capability differences. The benchmark is no longer discriminating between skill levels; it is recording how thoroughly each model has absorbed HumanEval-style patterns.
The practitioner implication is direct: do not use HumanEval scores to choose a coding model. SWE-bench Verified, and particularly the harder SWE-bench Pro variant where every model drops roughly 20 points from their Verified score, provides the more reliable signal. The compression you see in the HumanEval tab is the benchmark informing you it has nothing left to say.
What This Means for Choosing a Model
The chart is designed to surface a trade-off that single-metric leaderboards obscure. Cost and speed are stable and measurable; benchmark performance is conditional on the test design. The useful question is not “which model is smartest?” but “which model performs best on tasks that resemble mine, at the cost and speed my use case can tolerate?”
For scientific or research reasoning (question answering over technical documents, literature synthesis, analysis of complex domain knowledge) Gemini 3.1 Pro leads on GPQA, but DeepSeek V4 Pro closes the gap at eleven times lower cost. For high-volume applications, that arithmetic typically dominates.
For code repair, multi-file reasoning, or agentic software development, Claude Opus 4.8 leads on SWE-bench Verified, though Gemini 3.1 Pro and DeepSeek V4 Pro are within seven points. Speed becomes relevant in interactive contexts: Opus 4.8 runs at around 40 tokens per second versus 250 for Gemini 2.5 Flash, and that latency difference is perceptible in real-time coding workflows.
For mathematical reasoning, GPT-5.4 and Gemini 3.1 Pro tie at the AIME top. DeepSeek V4 Pro offers a strong third option at a cost that changes the economics of running reasoning agents at scale.
In all cases, the benchmark to treat with scepticism is the one with the narrowest spread. When nine models cluster between 80% and 94%, the benchmark is no longer your most reliable guide to capability; and the chart is telling you to look elsewhere.
Methodology
Benchmark scores are vendor-reported or sourced from independently-run evaluations on standardised harnesses. GPQA Diamond uses the 198-question expert-validated subset from the Arora et al. 2023 benchmark. SWE-bench Verified scores reflect the Scale AI standardised evaluation harness, April–June 2026; vendor self-reported scores on proprietary harnesses can differ by up to 20 points for the same model. AIME 2025 uses the 30-problem February examination. HumanEval is the 164-problem Python generation benchmark from OpenAI. API costs represent blended pricing — (input + output price) ÷ 2 — at official list rates, July 2026; actual costs depend on input/output ratio. Speed data is median tokens per second from Artificial Analysis, July 2026. Harness and evaluation differences mean cross-provider comparisons are directional rather than definitive.
