AI-Powered Proof Generation Could Slash Software Development Costs

A new process for automatically verifying that software is correct, using an AI model to mathematically prove its safety, could significantly reduce the time and expense required to review code. The method, developed by Anthropic and its partner Theorem, is intended to make it possible to ensure software is safe without relying on human testers or reviewers.

Formally verifying software, or proving that it runs without error under all possible conditions, has historically been very expensive. For example, it required 20 person-years to verify the seL4 operating system kernel, and the practice has been reserved for critical applications like Airbus avionics software.

To make the process more practical, Anthropic's Frontier Red Team worked with Theorem to use the Claude model to generate the proof code. The model produced roughly 180 million lines of proof code for 123 open-source libraries in just a few weeks. The process involves translating crash tests into mathematical problems and analyzing the compiled machine code to produce proofs, which are then automatically checked by a program called Rocq. Researchers call the process 'fully autonomous' proof generation.

During the process of generating proofs, the model also found hundreds of bugs that conventional testing had missed. These bugs often only appear under specific conditions, such as with very large input files or a particular sequence of accesses. For one widely used PDF reader, the team found as many security bugs in one week as are typically reported for that software in an entire year, and in another case, a logical error that had been undetected for over a decade was found. Anthropic states that even when a proof ultimately fails, the attempt can still reveal real bugs.

However, the company acknowledges that 'formally verified' does not mean 'error-free.' The proofs are only valid under strict conditions, such as for a specific processor architecture, single-threaded programs, and inputs under 64 MB. In addition, the compute cost for generating proofs is high, with individual proofs running up hundreds of millions of tokens, though the company did not report total project costs. Newer models are becoming more efficient at the task, with Opus 5.5 solving half the proofs with about 40 percent fewer tokens than Opus 5.

If an AI can write code and prove its safety, developers can trust the software more and spend less time reviewing it, which could reduce costs across the entire coding process. Anthropic expects that formal verification with AI will soon become as practical as standard testing, and Theorem's roadmap targets critical infrastructure like glibc, OpenSSL, and cURL. This approach could make it possible to automate more of the software development process.

The ability to automatically generate code and prove its safety could have significant implications for software development. If an AI can verify its own code, the question shifts from whether AI can program to whether it can prove that the code it writes is safe. If the answer is yes, this could lead to a major increase in software automation. Anthropic is working to make formal verification with AI a routine part of the software development process, just as standard testing is now.