Coding agents can write and run tests, but a new evaluation suggests they still struggle with the harder task: selecting meaningful properties, adversarial cases and verification targets that expose real defects.
Dan Luu tested coding agents on a Rust implementation of Zstd under 26 prompt conditions, ranging from no added instruction to requests for test-driven development (TDD), fuzzing, property-based testing, differential testing, formal methods and specific tools including Lean 4, Kani, Verus and QuickCheck. The evaluation also tried several reusable “skills” intended to guide testing behavior.
The broad result was not that one verification method won decisively. It was that explicitly naming a sophisticated method rarely made agents use it well.
Defaults held up better than many specialized prompts
Across runs using Codex with GPT-5.6 Sol at medium and extra-high effort settings, the default condition—no additional testing instruction—performed above average. At the higher effort setting, fuzzing and property-based-testing-related conditions did somewhat better on average than formal-methods conditions, though results were mixed and the author cautions against drawing strong conclusions from individual rankings.
TDD did not perform well. Testing skills recommended by Codex also underperformed, while a smaller custom skill performed reasonably. Luu’s interpretation is practical: the better prompt was designed to push the model away from unproductive default behavior, whereas the other skills read more like long tutorials.
That distinction matters for engineering leaders building internal agent workflows. A large knowledge artifact is not necessarily an effective operational instruction. If an agent cannot translate general testing guidance into a targeted test plan, adding more prose can consume context without improving outcomes.
The failure is in test design, not merely test execution
The agents were generally able to make the tests they wrote pass, particularly at the higher effort level. But passing self-authored tests is a weak proxy for correctness when those tests miss the meaningful failure modes.
The study reports recurring patterns: formal-methods prompts led agents to prove irrelevant properties; property-based-testing prompts led to indiscriminate random inputs, excessive invalid or rejected cases, or trivial assertions; and conventional tests often reused overly similar fixtures. In one cited class of mistake, a feature involving four bitstreams was tested with four identical streams—concealing bugs such as transposed inputs.
In other words, agents appeared able to adopt the vocabulary and tooling of verification without reliably applying the reasoning that makes those techniques valuable. The results were reportedly similar in smaller experiments on IMAP and other RFC-based tasks, not only the compression-format assignment.
What teams should do now
For teams using agents in production codebases, the operational implication is to treat agent-authored tests as implementation artifacts that require review, not as independent evidence of correctness.
A stronger workflow is likely to separate implementation from validation. Have one agent implement a change; have another, or a human reviewer, derive failure modes from the specification and inspect whether the test suite distinguishes them. Hidden or independently maintained test cases, differential comparisons against established implementations, and deliberately varied fixtures can provide more credible signals than asking an agent to “use TDD” or “do property-based testing.”
The benchmark also reinforces the value of specifying concrete invariants and known edge cases in task prompts. “Validate stream ordering against independently varied inputs” gives an agent a testable target; “use formal methods” may only give it a label.
What to watch next
The open question is whether model providers train agents specifically to design high-value tests. Luu argues that test-generation and verification tasks look amenable to reinforcement-learning environments, much as bounded performance optimization has been. Better training could improve this capability, but today’s evidence suggests that higher effort alone mainly helps agents complete their chosen approach—not necessarily choose the right one.
For buyers of coding-agent tools, benchmark claims should therefore be read carefully: code completion, test pass rates and tool use do not establish that an agent can verify its own work.
