OpenAI is publicly challenging one of the AI industry’s most watched coding benchmarks, arguing that a substantial share of tasks in SWE-Bench Pro may be flawed enough to distort model comparisons. In a new analysis published by OpenAI, the company said its audit found evidence that about 30% of tasks in the benchmark are broken.

The finding matters beyond one dataset. Coding benchmarks such as SWE-Bench Pro have become a shorthand for judging progress in agentic software development, shaping research priorities, launch messaging, and in some cases enterprise perceptions of which model is best suited for coding work. OpenAI’s core point is that if the underlying tasks are unreliable, headline scores may say less about actual software engineering ability than many model makers and buyers assume.

Why OpenAI is disputing a benchmark the industry has leaned on

According to OpenAI, the company began reviewing SWE-Bench Pro after previously concluding that SWE-bench Verified had significant design and contamination problems. At that time, OpenAI said it had encouraged the broader community to move toward SWE-Bench Pro as a stronger test of longer-horizon, more realistic coding tasks.

That newer benchmark has been influential partly because it appeared to show rapid progress. OpenAI wrote that on the 731-task public split, frontier models improved from a 23.3% pass rate to 80.3% in eight months. That kind of jump would normally suggest major gains in practical coding ability.

OpenAI now argues that those results need more caution. In its latest audit, the company said its internal datapoint analysis pipeline flagged 200 tasks, or 27.4% of the dataset, as broken. A separate human annotation campaign identified 249 tasks, or 34.1%, as broken. Based on those reviews, OpenAI said it estimates that about 30% of SWE-Bench Pro tasks are flawed.

The company’s position is not that every task is unusable, nor that no signal remains. Instead, it is warning that benchmark results should be interpreted carefully because a significant portion of the test may punish correct behavior or reward incomplete fixes.

The specific failure modes OpenAI says it found

OpenAI’s analysis breaks the problems into several categories that will be familiar to anyone who has worked with evaluations, especially in software tasks where hidden tests stand in for human judgment.

One category is overly strict tests. According to OpenAI, some tasks require a very specific implementation detail that is not stated in the prompt. That means a model can produce a functionally correct solution and still fail.

A second category is underspecified prompts. In those cases, OpenAI said hidden tests enforce requirements that are not reasonably inferable from the task description. This creates a mismatch between what the model is asked to do and what it is graded on.

A third issue is low-coverage tests. OpenAI said some tests do not adequately check whether the requested feature has actually been implemented, allowing partial or incomplete fixes to pass. For buyers evaluating AI coding systems, that is the reverse failure mode from overly strict tests: the benchmark can overstate capability, not just understate it.

OpenAI also pointed to misleading prompts that direct models toward behavior that conflicts with what the tests later expect. The company included an example involving Markdown table-of-contents formatting where the visible prompt and hidden test cases differed by a leading space. In that scenario, a model following the prompt as written could still be marked wrong.

That example gets to the heart of the broader critique. Many software issues and pull requests originate in real repositories for human collaboration, not as clean evaluation tasks. OpenAI argues that when those artifacts are converted into a benchmark, the prompt, reference patch, and tests do not always line up well enough to support reliable scoring.

How the audit was conducted, and where uncertainty remains

The strongest facts in this story come from OpenAI’s own published methodology, but they are still vendor-reported findings rather than an independent third-party benchmark review.

OpenAI said it built a quality assurance pipeline that examined task instructions, model attempts, metadata, and failure traces to identify likely problem cases. The company said an initial automated filter flagged 286 potentially broken tasks for deeper review.

From there, OpenAI used two parallel review paths. One involved Codex-based investigator agents with access to the repository and execution environment, allowing them to inspect files, run tests, and analyze model failures. The other was a human annotation campaign involving experienced software engineers. OpenAI said each flagged task in that review process was examined by five engineers, with disagreements escalated.

OpenAI reported that human reviewers were generally more likely than the agent pipeline to mark tasks as broken. The company also said category overlap between reviewer judgments and the agent pipeline was 74% for the flagged tasks, and that in no flagged task was “not broken” the most common human label.

Those details strengthen the company’s argument that the issues are not limited to isolated edge cases. But there are also limits to what can be concluded from the published material. The available source evidence does not include an external response from the maintainers of SWE-Bench Pro, nor does it provide a side-by-side re-scoring of major models under a repaired version of the dataset. So while OpenAI makes a substantial case that the benchmark is noisy, the exact impact on model rankings is still unclear from the evidence provided.

Why this matters for model builders and enterprise buyers

For AI labs, the immediate implication is that benchmark wins in coding may be less durable than they look. If nearly a third of tasks contain defects, leaderboard gaps could reflect prompt-test mismatches, implementation assumptions, or weak test coverage rather than real differences in model skill.

That matters especially for teams building AI agents for software work. SWE-Bench Pro has been positioned as a meaningful test of longer-horizon coding behavior, closer to what agent systems do in production. If its tasks are materially flawed, then builders need broader evaluation stacks that include repository-specific testing, human review, regression analysis, and real deployment metrics rather than a single public score.

For enterprise AI buyers, OpenAI’s warning is a reminder not to equate benchmark pass rates with production readiness. A model that performs well on SWE-Bench Pro may still struggle with reliability, ambiguity, or test completeness in a real codebase. Conversely, a model that appears weaker on the benchmark could be penalized by defects in the tasks themselves.

This is also a governance issue. OpenAI explicitly tied benchmark quality to safety and deployment decisions under its Preparedness Framework. The argument is straightforward: if evaluations overstate or understate capability, they can distort both product launch choices and risk assessments. That point extends beyond OpenAI and beyond coding benchmarks. As models are increasingly evaluated for agentic behavior, poor benchmark hygiene becomes a strategic problem, not just an academic one.

The company’s emphasis on agent-assisted auditing is notable too. OpenAI is effectively arguing that the same class of systems being tested can also help debug the tests. Whether that approach becomes standard will depend on whether outside researchers accept that agent-based review can improve dataset quality without introducing another layer of model-dependent bias.

Evidence, claims, and what is vendor-reported

The central claim in this story is vendor-reported: OpenAI says about 30% of SWE-Bench Pro tasks are broken based on its internal audit process and a human review campaign it organized.

OpenAI also reports that frontier models improved from 23.3% to 80.3% on the benchmark’s 731-task public split over eight months. In this article, that figure should be read as a benchmark trend cited by OpenAI, not as an independently verified market measure.

The methodological details are more concrete. OpenAI says its automated filter flagged 286 tasks, its datapoint analysis pipeline judged 200 tasks broken, and its human annotation campaign judged 249 broken. It also says five experienced engineers reviewed each flagged task and that reviewer judgments overlapped with the agent pipeline in 74% of flagged cases.

What remains unverified from the available evidence is how benchmark maintainers or other labs would classify the same tasks, whether repaired tasks would materially reorder model leaderboards, and whether similar error rates appear in adjacent coding benchmarks besides SWE-bench Verified and SWE-Bench Pro.

What to watch next

The first signal to watch is whether the maintainers of SWE-Bench Pro publish a response, revised dataset, or formal rebuttal. If they agree with a meaningful share of OpenAI’s findings, the benchmark could be updated quickly. If they dispute the conclusions, the field may need an independent adjudication process.

Second, watch whether major labs change how they report coding performance. If future model launches place less emphasis on SWE-Bench Pro and more on private eval suites, repository-level testing, or task-completion studies, that would indicate weakening confidence in public coding leaderboards.

Third, keep an eye on whether OpenAI releases more tooling or methodology around benchmark auditing using Codex or related investigator agents. A reproducible audit pipeline could influence how the industry validates not just coding datasets but also evaluations for AI agents more broadly.

Finally, the larger question is whether benchmark inflation is now a recurring pattern in fast-moving model categories. OpenAI previously criticized SWE-bench Verified, and now it is raising issues with SWE-Bench Pro. If the next generation of coding evaluations runs into similar problems, the market may shift toward mixed evidence: public benchmarks, customer telemetry, and controlled real-world workflow tests rather than single-number rankings.

Creati.ai perspective

OpenAI’s analysis is important less because it attacks one benchmark and more because it exposes a structural problem in AI measurement. The industry wants clean, comparable numbers for coding models, but software tasks are messy. When prompts, hidden tests, and reference patches are pulled from real repositories, small inconsistencies can quietly turn evaluation into noise. That is a problem for builders trying to optimize models and for enterprises trying to buy with confidence.

The practical takeaway is not to stop using SWE-Bench Pro, SWE-bench Verified, or other public benchmarks. It is to demote them from final verdicts to directional inputs. For teams shipping AI agents, benchmark literacy is now part of product work: understand what a score actually measures, what it misses, and how fragile it may be. The labs that earn trust in coding will be the ones that pair strong public results with transparent evaluation design, real workflow evidence, and a willingness to audit their own measuring sticks.