SWE-bench went from 60% to nearly 100%. That's the benchmark dying, not progress.
When a benchmark climbs from the 60s to nearly 100%, the instinct is to celebrate progress. The more honest reading is that the benchmark just died. That's roughly what happened to SWE-bench Verified, the number every AI coding tool has been quoting at you, and the story of how it broke is a useful lesson in why you should stop trusting leaderboards and start building your own.
OpenAI walked away from its own favourite metric
In February 2026, OpenAI published a post saying it would stop treating SWE-bench Verified as a measure of frontier coding ability. That's a striking thing for a lab to say about the benchmark it helped popularise. Their reasons, from the post, are the whole argument in miniature:
- Saturation. The top score had crept from 74.9% to 80.9% over six months, everyone bunched within a point or two of everyone else. When the whole field scores the same, the test has stopped discriminating.
- Flawed tasks. They audited 138 hard problems their model kept failing, and in 59.4% of them the test design or the problem statement itself was broken. The model wasn't wrong; the exam was.
- Contamination. A model solved tasks classed as nearly impossible, and in one Django case surfaced a parameter value that never appeared in the problem statement. It hadn't reasoned its way there. It had seen the answer before.
The memorization problem, measured
That last point is the one that should make you suspicious of every SWE-bench percentage you've ever read. A peer-reviewed paper with the excellent title "The SWE-Bench Illusion: When State-of-the-Art LLMs Remember Instead of Reason" put numbers on it. Models could identify the buggy file path from the issue description alone, with no access to the repository, at up to 76% accuracy. Point them at repositories outside the benchmark and that dropped to around 53%. The gap is the sound of memorization: the benchmark's answers leaked into training data, so a chunk of that impressive score is recall, not problem-solving.
A score of 90% on a contaminated benchmark and a score of 90% on genuine reasoning look identical on a slide. The whole point of an eval is to tell them apart, and this one stopped being able to.
The successor already shows the drop
There's a contamination-resistant follow-up, SWE-bench Pro: 1,865 tasks across 41 actively-maintained repositories, with a private, held-out split specifically so the answers can't leak. On today's public Pro leaderboard the same class of frontier models that near-saturate Verified sit in the low 60s and below. (Be careful quoting the "collapse" numbers, though: when Pro launched in 2025 the leaders were around 23%; models have since climbed into the 60s, so cite the era you mean.) Either way the shape is clear, harder, cleaner test, much lower scores, real spread between models again. That's what a working benchmark looks like.
What I actually take from this
Not "benchmarks are useless." The lesson is narrower and more practical: stop making decisions on a public leaderboard number. A headline SWE-bench score tells you how a model does on a public, likely-contaminated, possibly-flawed test that the whole industry has been optimising against for two years. It tells you almost nothing about how the model will do on your closed codebase with your conventions and your definition of "correct."
So the honest move, and the one I'd argue for, is to build a tiny private eval on your own repositories, work you know the right answer to, that has never been on the public internet. A dozen real tasks from your own git history, scored by you, will predict a tool's usefulness for your job far better than a four-year-old benchmark in the low 90s. It's the same instinct as the productivity studies I keep coming back to: measure on your own work, because the number on the box was optimised for someone else's.