A standardized test set used to score and compare model capabilities.
Broad technical consensus on what this term means. Disagreements that invoke it are about the world, not the word.
Benchmarks are the field's exams, fixed question sets (knowledge, math, coding, reasoning) on which models are scored and compared. Headlines, marketing, and a surprising amount of capital allocation key off them.
Their weaknesses are as load-bearing as their scores: test questions leak into training data (contamination), labs tune toward the tests, and saturated benchmarks stop discriminating. Whether benchmark gains still track real economic usefulness is one of the sharpest live disputes on the capability question.