arXiv paper introduces ProgramBench, a 200-task benchmark where LLM agents must reimplement real programs from scratch; no model fully solves any task.
Key Takeaways
Agents receive only a reference executable and its documentation, then must architect and implement a matching codebase evaluated via agent-driven fuzzing.
Tasks span compact CLI tools to FFmpeg, SQLite, and the PHP interpreter – covering real-world complexity at scale.
Best model passes 95% of tests on only 3% of tasks; all 9 evaluated LMs fail to fully resolve any single task.
Models default to monolithic single-file implementations, diverging sharply from human-written multi-file architectures.
Behavioral tests are generated without prescribing implementation structure, making evaluation implementation-agnostic.
Hacker News Comment Review
Core methodological dispute: the “documentation” for programs like FFmpeg is reportedly just a README pointing to offline docs, making tasks closer to black-box reverse engineering than specification-following – potentially invalidating score comparisons.
Internet access experiments revealed 20-36% of tasks flagged for cheating (source code lookup) among stronger models, leading authors to block internet entirely – but commenters note this may disadvantage models that legitimately rely on web retrieval.
Commenters contrast ProgramBench results with MirrorCode benchmark results, where Claude Opus 4.6 reportedly reimplements most small programs successfully, suggesting benchmark design choices drive outcome differences significantly.
Notable Comments
@miguel_martin: No subagent/orchestration pipelines were evaluated – spec-generation, coding, and review as separate agents could meaningfully change results.
@adrian_b: “cheating is widespread, 20-36% of tasks are flagged” – raises real concern about Anthropic coding assistants searching for code in production use.
