Real-world benchmark finds coding agents still require human cleanup on most tasks

AI coding agents can complete much of a software change while still failing to deliver a fully working result, according to a new benchmark built from real user requests. The test found that even the strongest system completed every required part of a task in only 26.4% of cases, meaning human cleanup remained necessary on nearly three out of four assignments.

The benchmark, called Co-Design Bench, was developed by researchers from Alibaba and several Asian universities. As reported by The Decoder, it evaluates agents on changes to existing web applications rather than isolated programming problems. The tasks can include visual references, audio, video, and documents, reflecting the mixed inputs that product and engineering teams increasingly give to AI tools.

A harder test for coding agents

Co-Design Bench is designed to address what its researchers describe as saturation in established coding evaluations. When models can execute, inspect, and repair their own code, they can score above 99% on some older benchmarks, leaving little room to distinguish the systems.

The new test instead draws on requests from actual coding sessions. The researchers selected cases in which users had already needed to rephrase requests or roll back unsuccessful changes. These range from relatively narrow jobs, such as extracting an amount from a receipt and updating related totals, to broader application redesigns.

That design makes the benchmark less representative of a clean, self-contained coding exercise and more representative of the messy work involved in maintaining a live product. It also means the results should not be read as a universal measure of every coding workflow. They do, however, highlight the gap between producing plausible code and completing a user’s full request.

The evidence behind the 26.4% result

According to the paper described by The Decoder, GPT-6 Astra was the top performer among the ten models tested. It satisfied roughly 70% of task requirements on average, but fully solved only 26.4% of tasks. A task counted as complete only when all required features worked in the application.

Claude Opus 5 and Qwen3.8-Max followed GPT-6 Astra in the reported ranking, while Grok 4.6 placed last. On larger application overhauls, the smaller Qwen3.8-27B model rose to second place, although the researchers did not treat that result as evidence of a general advantage for the model.

The grading process was also stricter than a simple code review. One automated judge tested the completed application in a browser, while a second reviewer checked positive results and looked for visual mock-ups, hardcoded outputs, or other shortcuts that could appear to satisfy a request without implementing the underlying behavior.

These are research findings, not independently verified adoption or productivity measurements. The benchmark’s authors acknowledge that its automated judges can make mistakes, and the tasks use one fixed request rather than a live conversation in which a user can answer questions or correct an agent midway through the work. The results therefore show where agents struggled under this test setup, not the precise amount of human labor required in every production environment.

The model is only part of the system

One of the benchmark’s most important findings concerns the agent harness: the software environment that lets a model read files, modify code, run applications, and inspect results. In a separate comparison, Qwen3.8-27B scored 75% on larger overhauls in one harness but only 56% in another. The relative ranking of two models also changed depending on the environment.

That result challenges the practice of comparing coding agents solely through model leaderboards. A model’s performance depends on the tools it can call, the context it receives, how it observes browser behavior, and how effectively it can recover from an incorrect change. For buyers evaluating an AI coding assistant, the practical unit of comparison is therefore the model-and-harness combination.

The study also found that extra computation did not reliably close the gap. GLM-5.2 used nearly six times as many tokens as GPT-6 Astra on smaller changes while achieving a significantly weaker score. More reasoning or a longer run may help on some tasks, but the result suggests that additional token usage is not a substitute for better task decomposition, testing, and interaction with the application.

What the findings mean for builders and enterprises

For engineering teams, the immediate implication is that AI coding agents remain most useful as supervised implementation partners rather than autonomous owners of product changes. An agent may create the bulk of a feature, but teams still need checks for edge cases, visual fidelity, data handling, and interactions between requirements.

The risk is particularly high for tasks that combine layout, design, interaction, and content changes. The benchmark found performance declined across models as more types of changes were combined, even though those intertwined requirements are common in real product work. A workflow that treats each request as a single prompt may therefore produce a convincing partial implementation while leaving important details unfinished.

Teams assessing tools should test them against representative internal tasks, including changes to existing codebases and applications rather than only greenfield examples. They should measure full acceptance, regression rates, review time, and the frequency of manual fixes. This complements established evaluations such as SWE-bench Verified, but it should not replace human review or deployment safeguards.

The findings also matter for agent-platform developers. Better browser inspection, clearer state tracking, reliable rollback, and stronger validation may improve outcomes as much as a larger underlying model. The harness result indicates that product teams should evaluate the entire workflow, including permissions, tool reliability, observability, and failure recovery.

What to watch next

The next useful signal will be whether Co-Design Bench or similar evaluations expand beyond fixed one-shot requests. Real software work involves clarification, changing requirements, and feedback from users or project leads, so interactive versions of the test could produce a different picture of agent usefulness.

Researchers and buyers should also watch for results that report model and harness together, rather than publishing a single model leaderboard. Independent replication would help establish whether the reported ranking and the large environment effects persist across codebases and application types.

Finally, acceptance rates in real engineering teams will remain more meaningful than benchmark completion alone. The Decoder cited earlier METR research showing that project leads accepted AI-generated code at a rate roughly 24 percentage points below scores on SWE-bench Verified. That gap is a warning that apparent task completion does not automatically translate into deployable software.

Creati.ai perspective

Co-Design Bench matters because it measures the last mile of coding-agent work: not whether an agent can produce code, but whether the requested product change actually works across the application. Its 26.4% full-completion result is a reminder that partial automation can still create substantial review and repair obligations.

For AI builders and enterprise buyers, the practical lesson is to stop treating the model as the whole product. Reliable execution will depend on the combination of model, harness, tests, application context, and human approval. Until those components are evaluated together on realistic tasks, claims of autonomous software development should be treated cautiously.