Why benchmarks exist and what they hide
LLM benchmarks are standardised tests that let people compare models on the same tasks with the same scoring. They are genuinely useful: without them, every claim about a model being “better” would be anecdotal. But a benchmark score is a summary statistic, and like any summary it hides as much as it reveals. A single number averages over hundreds or thousands of questions, some of which matter for your use case and most of which probably do not. The goal of this guide is to help you read the common benchmarks for what they actually measure, so a leaderboard becomes a starting point for your own evaluation rather than a verdict you accept blindly.
Knowledge and reasoning benchmarks
The most cited general benchmark is MMLU, a 57-subject multiple-choice exam covering everything from elementary mathematics to professional law and medicine. It measures broad knowledge and basic reasoning, but top models now score so highly that it has largely saturated — the differences between leaders are within noise. To restore difficulty, newer benchmarks like GPQA (graduate-level science questions designed to be “Google-proof”) and MMLU-Pro raise the bar with harder, less memorisable questions. For commonsense and reading, HellaSwag and ARC test whether a model picks plausible sentence continuations and answers grade-school science questions. When you see a knowledge score, ask whether the benchmark is still hard enough to separate the best models.
Coding, math, and conversational benchmarks
For code, HumanEval is the classic: write a short function that passes hidden tests. It is easy to game and narrow, so the field has moved toward SWE-bench, which asks models to fix real GitHub issues across full repositories — a far better proxy for actual software work. Math reasoning is measured by GSM8K (grade-school word problems) and the much harder MATH dataset (competition-level problems). For open-ended conversation and instruction-following, MT-Bench and the crowd-sourced Chatbot Arena (where humans vote between blind model responses) capture qualities that automatic tests miss, like helpfulness and tone. Each of these targets a different real-world skill, so a model can top one and trail on another.
Reading scores without being fooled
Two traps distort almost every leaderboard. The first is contamination: if a benchmark’s questions leaked into training data, the model may be recalling answers rather than reasoning them out, inflating its score. The second is saturation: above roughly ninety percent, remaining differences are mostly noise. Holistic frameworks like Stanford’s HELM try to counter single-number tunnel vision by reporting many metrics together — accuracy, calibration, robustness, fairness, efficiency — so you see trade-offs rather than one ranking. The practical takeaway is the same regardless of benchmark: use public scores to build a shortlist, then run a small, private evaluation on tasks that look like your real workload. Your own twenty representative prompts will tell you more than any leaderboard.