99% vs 26.4%: The Real Capability Boundary of Coding Agents
The gap between two numbers
Alibaba's new coding-agent benchmark, Co-Design Bench, delivers a striking contrast. On a test set built from real user requests, the top-performing model, GPT-6 Astra, fully solved only 26.4% of tasks — meaning nearly three out of four still required human cleanup. On older benchmarks, the same models score above 99%.
The design of the benchmark matters. Its tasks are not synthetic problems written from scratch; they are real requests that had already tripped agents up in practice. When a benchmark sources its questions from the scene of past failures, a lower score is not a regression in the model — it is the re-exposure of failures that were previously hidden.
Why old benchmarks break down
A 99% score and a 26.4% score can both be true only if they measure different things. Synthetic tasks tend to have clean boundaries, self-contained context, and a single acceptance criterion, letting a model probe a closed hypothesis space until it passes. Real requests are the opposite: requirements may be ambiguous, codebases carry historical baggage, and acceptance criteria are split across users, teams, and runtime environments.
The direct implication for engineering teams: public leaderboards are rapidly losing value as a selection tool. If a procurement or technical lead reads 99% and expects a workflow that needs almost no human intervention, 26.4% is the line they will hit in production.
The other side of the weekend project
In the same week, Andrej Karpathy demonstrated a different use. Using Claude's Opus 5.5, he filtered 43 million Hacker News comments into 6,454 curated videos, building a curation platform over a weekend and claiming it beats YouTube's own recommendations.
The two cases do not contradict each other; the difference is task structure. In Karpathy's project, a human defined the goal, the evaluation standard, and the trade-offs, while the model handled large-scale, splittable, spot-checkable filtering labor. Co-Design Bench, by contrast, tests end-to-end delivery. Agents are already usable in high-throughput intermediate stages, and remain fragile in the final stage where responsibility for correctness sits.
Sutton's reminder: compute is not the answer
Turing Award winner Richard Sutton reaffirms that raw compute alone is not enough for intelligence, and that fundamentally new learning methods are needed. Former OpenAI researcher Tworek puts it more sharply: blind transformer scaling "has reward hacked brains of a whole generation of scientists."
This discussion and the 26.4% figure are two sides of the same coin. When benchmarks saturate at 99%, the field can mistake that for general capability being in place; swap in the real distribution and the gap appears immediately. Meanwhile, in a Nature commentary, consciousness researchers Anil Seth, Ned Block, and Giulio Tononi argue that today's AI systems are not conscious and will not become so on silicon hardware, calling the assumption that consciousness depends on computation alone "fatally flawed." That debate need not map directly onto engineering choices, but it is a reminder: do not use anthropomorphic intuition to estimate an agent's reliability boundary.
Action items for teams
- Build a small real-world set of your own. Pull 30–50 requests from your product's failure tickets over the past three months and use them as an internal benchmark. It will predict your actual rework rate better than any public leaderboard.
- Split work by task structure, not by model. Separate the workflow into spot-checkable intermediate stages (bulk filtering, first drafts, retrieval and summarization) and accountable final stages. The former can largely go to agents; acceptance in the latter must stay with humans.
- Put human cleanup cost on the scorecard. Pass rates alone overstate the benefit; record the human editing time each agent output requires to get the real ROI.
- Stay skeptical of 99%. When a benchmark is near perfect, first ask where its questions came from and who defined acceptance.
The signals suggest the industry is moving from capability demos to reliability accounting. That shift is good news for people doing the deployment work: a real gap that has been quantified can be narrowed through engineering.
Sources
- AI coding agents still need human cleanup on nearly three out of four tasks in a real-world test — The Decoder
- The inventor of the bitter lesson has a bitter lesson for LLM scaling enthusiasts — The Decoder
- Andrej Karpathy built a curated video platform in a weekend using Claude and 20 years of Hacker News — The Decoder
- The brain-as-computer metaphor shaped modern AI but may be misleading the consciousness debate — The Decoder
Subscribe to weekly briefing