Benchmark
A fixed set of test questions with a scoring method, used to compare models on one kind of skill.
A benchmark might be thousands of multiple-choice exam questions, coding problems with hidden tests, or maths competition questions. Labs report scores on popular benchmarks at every release, which makes comparison easy but invites optimising for the test. Benchmarks age quickly as models saturate them or questions leak into training data, so new and harder ones keep appearing. Scores are best read as rough signals and only compared when the setup, such as the number of examples, is the same.