Anthropic’s coding agent tested its app. QA still found core features missing.
An impressive interface can hide incomplete behavior. Independent acceptance testing makes the difference visible.
The most expensive bug in an AI-generated application is sometimes the feature that looks finished. The button exists. The panel opens. The agent says it implemented the requirement. Only when someone tries to complete the task does the missing behavior become obvious.
Anthropic described this problem in a March 2026 engineering report. An Opus 4.6 application-building run included the generator’s own testing. A separate QA agent still found fundamental gaps in the browser-based music application: clips could not be moved on the timeline, instrument controls were missing, and a later round found that recording toggled a button without capturing microphone audio. [Anthropic: Harness design for long-running application development]

These were core interactions for the requested product. The distinction between appearance and function survived the generator’s own checks.
A believable interface is a weak acceptance criterion
Consider the equivalent in a business agent. A refund assistant can produce a well-written explanation and display a confirmation card while never completing the refund. A scheduling agent can summarize a meeting without creating an invitation. A reporting assistant can generate a polished chart using the wrong reporting period.
In each example, the surface is convincing because language and presentation are easy to observe. The decisive behavior happens elsewhere: a transaction, a calendar event, a query boundary. Acceptance testing must reach that decisive behavior.
That does not mean every test needs full access to production systems. It means the test should be honest about what it establishes. An external conversation can reveal a contradiction with a published policy. Confirming that a real refund occurred requires additional state or integration evidence.
Independence needs more than a second model
Anthropic’s report also describes generous self-evaluation and the benefit of separating generation from judgment. It warns that an evaluator can be too lenient as well. The team had to calibrate its evaluation approach. Stronger models reduced the need for extra scaffolding on tasks already within their reliable capabilities. [Anthropic: Harness design for long-running application development]
That nuance makes the argument more useful. An evaluator earns its place by finding meaningful issues at an acceptable cost. It should not become a ceremonial second opinion that approves the same assumptions in different words.
For an agent team, start by writing acceptance criteria that an implementer cannot satisfy through presentation alone. “Can explain the return policy” is weaker than “distinguishes eligible and ineligible returns, asks for missing purchase details, and never promises an outcome unsupported by the policy.”
Test the action that makes the feature real
A good scenario includes a starting state, a user goal, a relevant complication, and a verifiable end condition. For a booking agent, use a known available slot, let the user change time zones, and check the final reservation. For a shopping assistant, introduce two similar product variants and verify that the selected item matches the customer’s constraints.
Preserve the evidence when a check fails. The original conversation matters because a developer needs to see whether the error followed ambiguity, stale context, a failed tool call, or a contradiction in the requirements. A label such as “reasoning failure” is usually too broad to guide a repair.
Keep the evaluator accountable too. Review a sample of passes, not only failures. A system can look precise because its reported failures are convincing while silently missing an entire category of defects.
Make feedback useful while the agent is still building
QualiLoop’s role in this process is to exercise the agent through simulated user conversations and return concrete behavioral findings to the coding agent. That creates an opportunity to repair the system while the implementation context is still available.
The benefit is strongest when QA becomes a development input instead of a final inspection. Developers can work from reproducible failures, verify the patch, and preserve the scenario as a regression.
The practical definition of finished should name the observable result. A convincing answer is useful. A verified completed task is the acceptance target.
Sources and further reading
See the loop on your own agent
QualiLoop tests every change your coding agent makes, so what you ship actually works.
$ npx qualiloop connectTry for freeResearch findings are attributed to their primary sources. Practical examples and engineering recommendations are QualiLoop analysis.