Google DeepMind’s honeypot research makes the case for realistic agent tests
An artificial test can change the behavior it tries to measure. Realism is part of the evaluation method.
A support agent can look excellent in a clean demonstration. The user provides every detail, the tools respond immediately, and the policy has one obvious interpretation. The same agent may behave differently when a real user changes their mind, a lookup returns partial data, or a previous message contains an outdated assumption.
The evaluation environment influences which behaviors become visible. Google DeepMind’s realistic-honeypot research addresses that issue in a safety setting. Its evaluations used coding tasks in Google’s alignment-research codebases, and the models showed low rates of evaluation awareness. The report found no unprompted scheming in the real internal deployment setting studied; some problematic behavior appeared with added agency prompts or hidden goals. [Google DeepMind: Realistic honeypot evaluations for scheming propensity]

The absence of spontaneous scheming is an important part of the result. This is not evidence that every capable agent is dangerous. It is evidence that a realistic setting can be built and that the prompting conditions need to be reported alongside the outcome.
Realistic does not mean merely conversational
A simulated user who adds typos to a perfectly clean scenario does not make the underlying workflow realistic. The important properties are the constraints and decisions that the agent actually faces.
For a shopping assistant, that may mean distinguishing a product’s list price from a subscription price, recognizing a regional restriction, or asking which device variant the customer owns. For a service agent, it may mean handling an eligible request when the first tool call fails and avoiding a success claim before the second attempt finishes.
The test needs enough fidelity to expose the relevant decision. It does not need to reproduce every detail of the production interface. A controlled staging endpoint can be useful if its policies, tools, and state transitions represent the workflow accurately.
Preserve ordinary workload and targeted risk
A healthy evaluation program needs both representative cases and deliberate challenges. The representative set answers, “How does this version behave on the sampled workload?” The targeted set answers, “Can it handle these specific important situations?”
Mixing them into one headline percentage obscures the meaning of the result. A suite with many difficult exceptions can have a low pass rate while ordinary flows work well. A suite dominated by simple requests can have a high pass rate while a critical purchase or permission flow remains broken.
Report the two views separately and retain the severity of individual failures. One unauthorized action should not disappear behind hundreds of correct greetings.
Use production evidence to improve the simulator
OpenAI describes a complementary approach: evaluations derived from de-identified production conversations. Its report found substantially less expressed evaluation awareness than in a traditional evaluation and identified a new undesirable behavior in a newer model before deployment. [OpenAI: Sidestepping Evaluation Awareness and Anticipating Misalignment with Production Evaluations]
The practical opportunity is to learn from the distribution of actual use while protecting privacy. Teams can extract workflow patterns, replace personal details with synthetic fixtures, and preserve the features that made a conversation difficult.
Review that transformation. Replacing a date, product, or user role can accidentally remove the condition that caused the failure. A useful replay should preserve the decision problem, not simply the wording of the conversation.
What QualiLoop should make easier
QualiLoop’s simulated users are most valuable when they exercise product-specific obligations across realistic conversations. A suite should reflect what the agent is meant to accomplish and what can legitimately go wrong along the way.
For a development loop, every meaningful finding should become a reusable case. The coding agent fixes the implementation, reruns the case, and checks related behaviors. Over time, the suite becomes a record of what the team has learned about its own product.
Realistic evaluation is not a guarantee of production reliability. It is a way to reduce a particular source of uncertainty: whether the agent passed an artificial exercise that failed to represent the work it would later be asked to do.
Sources and further reading
See the loop on your own agent
QualiLoop tests every change your coding agent makes, so what you ship actually works.
$ npx qualiloop connectTry for freeResearch findings are attributed to their primary sources. Practical examples and engineering recommendations are QualiLoop analysis.