OpenAI’s agent improvement loop reveals the problem with green tests
Five passing evaluations can still leave a weak validator. The next development bottleneck is the quality of the feedback loop.
A coding agent finishes a patch. The checks are green. It reports success, and everyone moves to the next task. That workflow feels complete because it resembles ordinary software development. But an agent can pass a narrow test while still failing the job the test was meant to represent.
OpenAI’s agent-improvement cookbook contains a useful example of this gap. Its financial-diligence agent passed all five generated evaluations. Yet the accompanying analysis still recommended stronger validators: existing checks could approve outputs with unresolved citation or claim-audit problems. It also recommended deterministic financial checks and keeping the generated cases as durable regressions. [OpenAI Cookbook: Build an Agent Improvement Loop with Traces, Evals, and Codex]

The lesson is specific: a passing result establishes that the implemented checks passed. Whether those checks adequately represent the requirement is a separate question.
The validator can share the implementation’s blind spot
Imagine a shopping assistant that recommends a replacement filter. A test checks whether the response contains the correct filter family. The answer passes, even though the assistant chose a size that does not fit the customer’s device. The test recognized a relevant product; it never checked compatibility.
Adding more conversations with the same weak assertion will produce more data without resolving that gap. The missing ingredient is a better definition of success: the selected SKU must be compatible with the identified model, available in the relevant market, and presented at the current price. When the model is unknown, the assistant should clarify rather than guess.
This is an illustrative engineering scenario, not a finding from OpenAI’s example. It shows why evaluation design is part of product development. A suite needs to encode the product’s obligations, not simply patterns that resemble good answers.
What the OpenAI architecture gets right
The cookbook connects traces, human and model feedback, generated evaluations, HALO diagnosis, and a handoff to Codex. HALO prioritizes changes; a coding agent or human implements them. The workflow can then repeat, with human review gates where the team wants them. [OpenAI Cookbook: Build an Agent Improvement Loop with Traces, Evals, and Codex]
That separation creates useful working artifacts. A trace says what happened. A finding explains why it conflicts with a requirement. A proposed change gives the implementer a starting point. A subsequent run tests whether the change helped.
The distinction matters when a problem looks easy to fix. “The assistant invented a price” does not establish that the prompt needs another warning. The source could be an outdated cache, a missing tool result, a currency conversion bug, or a fallback that treats incomplete retrieval as authoritative. The repair should follow the evidence.
Give the coding agent a repair packet
A useful repair packet should include the agent version, relevant user turns, tool calls and responses where available, the expected outcome, and the exact mismatch. It should distinguish an observed failure from a suspected cause.
For the filter example, attach the compatibility source and the incorrect SKU. Describe whether the assistant had enough information to make a selection. If the tool supplied a correct compatibility list and the agent ignored it, investigate selection logic. If the tool returned incomplete data, investigate the integration before changing the prompt.
After the patch, rerun the original failure and nearby variations. Then run the broader regression suite. A change that fixes one model family but breaks another is a tradeoff the team needs to see, not a success to hide inside an average.
Where QualiLoop fits
QualiLoop’s approach is to make this behavioral feedback available inside the coding workflow: simulated users exercise the agent, findings preserve conversation evidence, and the coding agent can use that evidence to implement and verify changes.
The standard to aim for is a shorter route from an ambiguous complaint to a verified repair. Measure that route directly: time to reproduce, time to identify a useful fix, and time to demonstrate that important existing behaviors still work.
OpenAI’s cookbook is an implementation example, not an independent validation of QualiLoop or a development-speed benchmark. Its architectural direction is nevertheless relevant: improvement becomes a repeatable engineering process when feedback, implementation, and verification remain connected.
Sources and further reading
See the loop on your own agent
QualiLoop tests every change your coding agent makes, so what you ship actually works.
$ npx qualiloop connectTry for freeResearch findings are attributed to their primary sources. Practical examples and engineering recommendations are QualiLoop analysis.