HumanEval
A 2021 coding benchmark of 164 Python problems, each checked by hidden unit tests.
HumanEval was released by OpenAI with its Codex paper in 2021. Each problem gives a function signature and a description, and the model's code is run against tests. Results are reported as pass@k, the chance that at least one of k attempts passes. Codex solved about 29 per cent with one attempt; leading models now solve well over 90 per cent, so it has largely been replaced by harder and more realistic tests such as SWE-bench and LiveCodeBench, the latter using fresh contest problems to avoid contamination.