Your coding agent can write the tests and still miss the failure
What Anthropic’s engineering experiments tell us about self-testing, and how to build a stronger definition of done.
A coding agent adds a cancellation feature, runs its checks and reports that the work is complete. The endpoint responds. The unit tests pass. In a real conversation, a customer changes their mind halfway through, the agent loses the order reference, and then says the cancellation succeeded without calling the tool.
The implementation passed the checks that were performed. The customer’s task still failed.
This is an illustrative example, but it captures an important distinction: a coding agent’s ability to make a change does not establish that it has adequately tested the behavior the change creates.
What Anthropic actually observed
In its November 2025 engineering report, Anthropic described Claude marking features complete without proper testing. Even when it ran unit tests or made requests to a development server, it could miss that the feature failed end to end. The team improved its harness by providing stronger testing support, including browser-based verification. Anthropic: Effective harnesses for long-running agents

Its March 2026 follow-up described another problem: agents tended to judge their own work generously. A separate evaluator helped, but needed tuning; early versions tested superficially and sometimes approved work despite identifying legitimate issues. Stronger models also reduced the need for evaluation scaffolding on tasks they already handled reliably. Anthropic: Harness design for long-running application development
These are engineering observations from particular setups, not proof that all coding agents fail at self-testing. They support a narrower and more useful conclusion: self-testing alone is a weak acceptance process for complex agent behavior.
The same assumption can survive both code and tests
Consider a support policy that allows cancellations until dispatch. A developer interprets the rule as “until the order is packed.” If the same interpretation drives both the implementation and the test assertions, the tests can pass while the product violates the policy.
This is our engineering interpretation, rather than a causal finding from those Anthropic experiments. It explains why verification needs an authoritative reference outside the implementation: approved policy, product requirements, a known expected state or another independently checkable outcome.
Using a different model does not automatically provide that independence. Two models can share the same mistaken specification. A separate system becomes useful when it makes the acceptance criteria explicit and checks behavior against them consistently.
Test the conversation and the action
For the cancellation example, a useful behavioral test needs more than a single question. Start with an eligible order, ask to cancel it, change a detail, ask for confirmation, and inspect the resulting order state.
The transcript shows what the agent told the customer. The tool result and final state show whether it did the job. Anthropic’s evaluation guidance makes this distinction explicit: an agent’s claim that it completed an action is different from the outcome in the environment. Anthropic: Demystifying evals for AI agents

Where state access is unavailable, record that limit. You can verify whether the answer follows a policy, but you may not be able to verify that a refund or cancellation actually occurred.
Define done before the agent declares it
A practical acceptance process has four parts: explicit expectations, realistic conversations, observable outcomes and regression checks. Ask whether the agent completes the task when the user changes details, when a tool fails and when required information is missing. Repeat important scenarios so one lucky response does not decide the result.
Protect those expectations from silent changes. If an implementation fails a legitimate test, rewriting the assertion to match the implementation is not evidence of a fix. Test changes can be correct, but they need a reason tied to the product requirements.
How we apply this at QualiLoop
QualiLoop learns the agent’s intended behavior from its prompts, tools and documentation, then generates simulated user conversations for a defined scope. Findings include the relevant messages, the expected behavior and a proposed fix. Recurring failures are grouped so the coding agent can address an underlying problem rather than receive a pile of duplicate complaints.
The coding agent remains responsible for implementation. QualiLoop supplies a repeatable verification process around its changes. Teams still need to review the initial expectations and calibrate judgments against expert decisions and observable outcomes.
The useful question is not whether a coding agent can write tests. It is whether the evidence is strong enough to accept the change.
See the loop on your own agent
QualiLoop tests every change your coding agent makes, so what you ship actually works.
$ npx qualiloop connectTry for freeSources and further reading
- Anthropic: Effective harnesses for long-running agents26 November 2025
- Anthropic: Harness design for long-running application development24 March 2026
- Anthropic: Demystifying evals for AI agents9 January 2026
Written by QualiLoop. The cited publications do not endorse QualiLoop or validate its performance claims. Engineering recommendations and illustrative examples are ours. Product screenshots show QualiLoop’s existing testing interface.