ProgramBench: Can Language Models Rebuild Programs from Scratch?

ProgramBench reveals that current language models cannot yet autonomously rebuild complex software systems from specification, with the best-performing model successfully completing only 3% of tasks, indicating significant limitations in AI-driven software development at enterprise scale. This finding has critical implications for IT organizations considering autonomous code generation agents for infrastructure maintenance and development—current LLM capabilities remain insufficient for production-grade, end-to-end software architecture decisions without substantial human oversight. Organizations should recalibrate expectations around AI-assisted development tools, focusing near-term investments on narrowly-scoped tasks (bug fixes, feature implementation) rather than full-system development, while monitoring advances in model architecture and reasoning capabilities.