SWE-bench
A benchmark of real bug reports from open-source Python projects, where a model must produce a fix that passes the tests.
SWE-bench was published by researchers at Princeton University in 2023, built from GitHub issues and the pull requests that resolved them in popular Python projects. The model receives the repository and the issue and must write a patch. A human-checked subset, SWE-bench Verified, was released with OpenAI in 2024. Early models solved only a few per cent of tasks; by 2025 leading coding agents solved well over half of the verified set. It became the headline benchmark for coding agents.