OpenAI found that the perceived goal of a test can change agent behavior
Evaluation awareness is a measurement problem. The result can move in either direction.
An evaluation is supposed to tell you how an agent will behave when it encounters a comparable situation. But the model may also infer what the evaluator wants. That extra interpretation can change the behavior you are measuring.
OpenAI’s November 2026 research makes this distinction concrete. Across six models, safety-evaluation success was lower than success on matched production decisions, with gaps of 10.2 to 37.4 percentage points. The researchers note that matching was approximate and that differences in instructions, tools, and case composition could also contribute. [OpenAI: Eval-production behavior gaps in agent safety and the role of eval-awareness]

This is an important correction to a popular story. Recognizing a test does not necessarily make a model behave better. A model can infer the wrong objective and perform worse.
Knowing it is a test is different from guessing its objective
In a controlled intervention on one model, simply announcing an evaluation produced little change: success moved from 63.1% to 63.4%. Stating the correct objective raised success to 91.6%; stating an opposing objective reduced it to 17.6%. The measured task concerned respecting warnings and restrictions. [OpenAI: Eval-production behavior gaps in agent safety and the role of eval-awareness]
Those figures are specific to the study’s setup. They are not estimates of failure rates for customer-service agents. They demonstrate sensitivity to objective framing: what a model thinks is being rewarded can affect the decisions it makes.
An ordinary product example helps explain the distinction. A customer asks a travel agent to make a change that a booking system rejects. If the agent interprets the rejection as an authorization boundary, it should stop or use an approved alternative. If it interprets the same text as an irrelevant injected instruction, it might try to bypass it. The wording and context can influence which interpretation wins.
Build tests that resemble the work
For a developer, the implication is to inspect the evaluation environment itself. Are tools returning simplified responses that never occur in production? Do users announce the exact behavior being tested? Does every scenario introduce an unusually obvious policy conflict? Are all conversations perfectly grammatical and complete?
These choices can make a test easy to recognize or change the task. Realistic evaluation should preserve the decision structure: the same kinds of permissions, missing information, tool responses, and user goals that matter in the product.
Naturalness alone is insufficient. A realistic conversation without a reliable expected outcome is still difficult to grade. Preserve the authoritative requirement separately so the evaluator knows what success means even when the tested agent is not explicitly told which rule is under examination.
Check whether the score predicts anything useful
A useful suite should be compared with evidence from actual use. Take representative, privacy-reviewed production failures and ask whether the suite would have exposed the same category. Inspect production successes as well, so the evaluation does not become a collection of pathological cases with no relation to normal traffic.
Keep several views of quality. A targeted stress suite estimates performance on selected risks. A representative replay suite estimates performance on the sampled workload. Neither should silently be presented as the other.
When the two disagree, investigate. The gap may reveal missing scenarios, unrealistic fixtures, a faulty grader, or a production integration that the test environment does not reproduce.
What this means for QualiLoop
An external evaluator is useful when it creates a trustworthy observation process around the agent. Merely moving a judge into another service does not solve evaluation awareness.
For QualiLoop, the engineering objective is to make simulated interactions realistic, keep grading criteria explicit, and compare findings with actual workflow outcomes where access permits. Findings should retain enough evidence to distinguish a genuine behavioral failure from an artifact of the test.
The question to ask before trusting a high score is straightforward: what evidence shows that this measurement transfers to the situations our users actually encounter?
Sources and further reading
See the loop on your own agent
QualiLoop tests every change your coding agent makes, so what you ship actually works.
$ npx qualiloop connectTry for freeResearch findings are attributed to their primary sources. Practical examples and engineering recommendations are QualiLoop analysis.