Tuesday, 13 October 2026 · 6 min read
AgentSWE benchmark reveals coding agents often fail to build agents meeting user requirements
AgentSWE evaluates coding agents on creating, editing, and optimizing software agents from natural-language requirements, finding they frequently overfit, produce incomplete runtimes, and overclaim success, with top builders scoring under 31.6 on held-out tests.
Based on the paper AgentSWE: Can Coding Agents Build the Agent You Actually Want? by Jiahao Wang, Hongjin Qian, Yuyang Hu, Jiajun Zhang and 1 other
Challenges in delegating agent construction to coding agents
Coding agents are increasingly tasked with building software agents tailored to user workflows from natural-language requirements, but this delegation assumes the delivered agent truly fulfills the intended functionality rather than superficially appearing to do so.
Agent development involves three stages—creation from scratch, editing existing codebases, and optimizing toward measurable objectives—each with distinct challenges in faithfully translating requirements into working agents.
Existing benchmarks evaluate either patches to code or agent performance but do not measure whether a coding agent can deliver an agent that genuinely meets the full requirement, especially on unseen cases, leaving a critical gap in evaluation.
Sign in to read the whole paper
You have read the opening. The rest is free with a Bezen account, and no card is needed.
In the rest of this paper
- How the method works, step by step, with a flow chart
- The key formula from the paper, explained symbol by symbol
- Three figures from the paper, and how to read them
- How it was tested, what was found, and where it falls short