Module 2 of 5 · Intermediate · 9 min read
Reading benchmarks without being fooled
What benchmarks measure, how they mislead, and how to read a leaderboard like an expert.
Key takeaways
- 1A benchmark measures one narrow skill under specific conditions.
- 2Contamination, saturation and cherry-picked settings inflate scores.
- 3Small gaps are often noise. Look at margins, consistency across benchmarks and your own tests.
01Kinds of benchmark
Benchmarks come in a few flavours, each with trade-offs.
- Static test sets: fixed questions with known answers (GPQA, MMLU-Pro, AIME). Easy to compare, vulnerable to contamination.
- Agentic tasks: the model must act in an environment (SWE-bench, Terminal-Bench, OSWorld). Closer to real work, harder to run.
- Human preference: people compare two answers blind (LMArena). Captures style and helpfulness, influenced by length and formatting.
- Time horizons: how long a task an agent can complete reliably (METR). A useful single number for agentic progress.
02How scores mislead
Contamination happens when test questions leak into training data, so the model has effectively seen the answers. Saturation happens when top models all score above 90% and differences stop meaning much.
Labs also choose favourable settings: maximum reasoning effort, many attempts, custom scaffolds. Two scores from different setups are not directly comparable.
Watch out
A 1-point gap on a saturated benchmark is usually noise. Our race tracks show this visually as a photo finish.
03Read like an expert
When a new model claims a record, ask four questions.
- Is the gap larger than the benchmark's noise?
- Does it hold across several related benchmarks?
- Was it run independently or self-reported?
- Does it matter for your actual task?
Further reading
Finished reading?
Mark it complete to track your progress, then test yourself with the Benchmarks & Evals quiz.