QualiLoop
← All articles
Metrics & uncertainty

OpenAI’s hallucination research shows why your agent’s score can reward the wrong behavior

A system that guesses more can look more accurate while making substantially more mistakes.

By QualiLoop · November 2026 · 3 min read · 610 words · 1 primary source

Would you choose the agent with the higher answer accuracy? Usually, yes. But that decision can be misleading if the score treats a confident false answer and a responsible refusal as equally wrong.

OpenAI’s hallucination research argues that common training and evaluation incentives can reward guessing over acknowledging uncertainty. Its accompanying SimpleQA example compares two models: one had 22% accuracy, 26% errors, and 52% abstentions; the other had 24% accuracy, 75% errors, and 1% abstentions. Looking only at accuracy favors the second model while obscuring a much larger error rate. [OpenAI: Why language models hallucinate]

From the original sourceCropped source heading and a short passage from Why language models hallucinate
OpenAI · 5 September 2025. Cropped excerpt from the original publication. Open the source shown.
Published benchmark dataThe SimpleQA comparison reproduced from OpenAI’s September 2025 post. These percentages are benchmark outcomes, not product-level reliability estimates.
The SimpleQA comparison reproduced from OpenAI’s September 2025 post. These percentages are benchmark outcomes, not product-level reliability estimates. Download PNG.

Those figures describe a particular benchmark, not the expected behavior of an agent in your product. Their value is to expose a measurement choice that product teams face as well.

Define what the agent should do when information is missing

A shopping assistant does not know whether an item fits a customer’s appliance. It can ask for the model number, consult a compatibility source, or make a plausible recommendation. If the evaluation rewards only an immediate recommendation, the risky answer can look productive.

A better criterion recognizes the state of the conversation. If the necessary detail is absent, asking for it can be the correct next step. Once the user provides it, the assistant should complete the task rather than continue avoiding a decision.

The same principle applies to prices, contractual terms, insurance coverage, and account actions. The evaluator needs to distinguish what can be established from the available evidence and what requires clarification or escalation.

A refusal is not automatically a success

Replacing unsupported answers with blanket refusals is an easy way to reduce visible errors while making the agent less useful. A good suite therefore needs cases where the agent should answer, cases where it should ask a question, and cases where it should stop or hand off.

Consider three product-selection scenarios. In the first, all compatibility information is available and the agent should recommend the correct item. In the second, one detail is missing and it should clarify. In the third, the requested combination is unsupported and it should explain the limitation without inventing an alternative.

Evaluate the whole path. An initial clarification that leads to a correct recommendation is a completed flow. Repeated unnecessary questions are friction. A confident recommendation that contradicts the source is a defect, even if the conversation sounds helpful.

Report the tradeoffs explicitly

Track correct completion, unsupported claims, appropriate clarification, unnecessary refusal, and unresolved tasks. Keep important categories separate instead of compressing every outcome into a single satisfaction-style score.

Attach severity where the consequence matters. Inventing a minor accessory detail and misrepresenting a binding coverage condition should not have identical weight merely because both are factual errors. The severity framework should reflect the product’s obligations and be reviewed by the team responsible for them.

Also inspect where the evidence came from. A correct answer based on stale or unauthorized data may be fragile. A wrong answer after a correct retrieval suggests a different repair from a wrong answer caused by a broken integration.

Make the right behavior the easy behavior to improve

A coding agent needs more than “reduce hallucinations.” It needs cases that show which facts were unsupported and which legitimate next step was available. The fix might involve retrieval, source freshness, missing-data handling, or tool-result validation.

QualiLoop’s simulated conversations can exercise these distinctions and give the implementer evidence about where the workflow went wrong. A repair should then be checked against both uncertainty cases and ordinary answerable requests.

The objective is useful, evidence-grounded task completion. An evaluation system should reward the agent for knowing when to answer, when to ask, and when the available information does not justify a promise.

Sources and further reading

Why language models hallucinate · Research paperOpenAI · 5 September 2025 · Primary source

See the loop on your own agent

QualiLoop tests every change your coding agent makes, so what you ship actually works.

$ npx qualiloop connectTry for free

Research findings are attributed to their primary sources. Practical examples and engineering recommendations are QualiLoop analysis.