The fastest route to a working agent runs through better feedback
Why evaluation can shorten development, how to measure the effect, and what the research does not prove.
An AI agent can take seconds to propose a patch. Finding out whether that patch solves a real user problem can take much longer.
A developer reproduces the failure, reads the conversation, checks a trace, explains the issue to a coding assistant, tries the new version and then checks whether anything else broke. The model may be fast throughout. The development loop is still slow.
Evaluation becomes a productivity tool when it reduces the work between a failed interaction and a verified improvement.
What OpenAI’s engineering experience adds
OpenAI’s February 2026 harness-engineering report described human QA capacity becoming a bottleneck as code throughput increased. The team made application behavior and observability accessible to Codex, and connected implementation with validation and feedback. This is evidence from one internal engineering effort, not a controlled estimate of the effect of any individual evaluation tool. OpenAI: Harness engineering: leveraging Codex in an agent-first world

The practical implication is that a coding agent needs useful feedback channels, not simply permission to keep generating changes.
Turn a complaint into a repair task
“The router sometimes chooses the wrong flow” leaves several questions unanswered. Which conversation? What history did the router see? Which destination was correct? Did the failure happen before or after a tool call?
A useful finding answers those questions before the coding agent starts editing. For example: after a user asks about a delivery and then says “cancel that instead,” the agent routes into product discovery. The conversation history identifies an active order, so the expected behavior is cancellation handling or an eligibility check.
This is an illustrative diagnosis. It demonstrates the information a repair loop needs: the triggering interaction, the expected decision, the observed decision and enough context to investigate the difference.
A proposed fix can then be tested as a hypothesis. Perhaps the router did not receive the history. Perhaps it received history but ignored the relevant entity. A transcript alone cannot distinguish those causes. Connected traces and code inspection can.

Why grouping failures can save investigation time
Twenty failed conversations may come from one broken handoff. Treating every transcript as an unrelated bug creates redundant investigation. Grouping them by a supported underlying cause gives the coding agent one repair task and several examples it must satisfy.
Grouping also needs restraint. Two failures with similar wording can have different causes. Keep the original evidence accessible and split a group when traces or code inspection show distinct mechanisms.
After a patch, rerun the failing cases first for fast feedback. Then run the broader regression suite before accepting the change. The first pass asks whether the specific repair helped. The second asks what else changed.
The speed claim needs the right denominator
At QualiLoop, we have observed up to 20x faster development in some internal experiments using the connected testing and repair workflow. This is our reported experimental result. It is not an Anthropic or OpenAI result, and it is not a forecast for every customer.
A meaningful measurement compares the time needed to reach the same independently verified acceptance criteria. Count setup, test execution, diagnosis, implementation, reruns and human review. Also record model cost and the rate of failures missed by the evaluator.
Setup + implementation + testing + diagnosis + repairs + verification. Compare the whole process at a matched quality bar.
Use the same model, starting repository and task set across the comparison. Repeat runs, preserve failures, and report the median as well as the best case. A 20x improvement on one difficult debugging task does not establish a 20x improvement across an entire engineering organization.
Make the loop selective
Evaluation has a cost. Anthropic’s follow-up harness work found that a separate evaluator could become overhead for tasks a stronger model already handled reliably, while still helping on harder work. That argues for testing proportional to the change and risk. Anthropic: Harness design for long-running application development
Use targeted checks during iteration, broader checks before release and deeper tests when tools, routing, policies or models change. OpenAI’s cookbook also presents an implementation pattern connecting traces, evaluations and a coding agent into an improvement loop. It establishes a useful architecture, not a universal speed guarantee. OpenAI Cookbook: Build an Agent Improvement Loop with Traces, Evals, and Codex
QualiLoop applies this principle to AI-agent development: generated user simulations produce evidence, the coding agent applies a change, and evaluation measures what improved. The commercial value comes from reaching a working system with less manual investigation.
See the loop on your own agent
QualiLoop tests every change your coding agent makes, so what you ship actually works.
$ npx qualiloop connectTry for freeSources and further reading
- OpenAI: Harness engineering: leveraging Codex in an agent-first world11 February 2026
- Anthropic: Harness design for long-running application development24 March 2026
- OpenAI Cookbook: Build an Agent Improvement Loop with Traces, Evals, and CodexAccessed 16 November 2026
Written by QualiLoop. The cited publications do not endorse QualiLoop or validate its performance claims. Engineering recommendations and illustrative examples are ours. Product screenshots show QualiLoop’s existing testing interface.