Hallucination Is a Governance Problem, Not Just a Model Problem

Every discussion of hallucination eventually arrives at the same hope: the next model will be better. Confabulation rates drop with each generation, the benchmarks improve, and it is tempting to treat the whole issue as a temporary defect that scale will iron out. In low-stakes work, that hope is even reasonable. If a model invents a detail in a brainstorming session, you notice and move on.

However, in high-consequence work the framing is wrong, and staying with it is dangerous. A probabilistic model does not "malfunction" when it confabulates. It is doing exactly what it was built to do: produce the most plausible continuation, whether or not that continuation is true. Plausibility and truth are correlated, not identical, and the gap between them does not close to zero at any model size we can currently see. So a system that decides on a loan, flags a transaction, or validates a compliance dossier cannot be made safe by waiting for a better model. The residual error is not a bug you are one release away from fixing. It is a property you have to govern.

This is the shift that matters. The question stops being "how do we get a model that never makes things up?" and becomes "how do we run a probabilistic system inside a process where being confidently wrong is not allowed to reach a consequence unchecked?" That is a governance question, and it has real answers.

Grounding: answer from the record, and cite it

The first control attacks the most common failure directly. A model asked to answer from its own weights will fill gaps with plausible invention. A model required to answer only from a supplied, retrieved record, and to cite the specific source for each claim, has far less room to confabulate, because every assertion has to point at something real.

Grounding does two things at once. It reduces the invention rate, because the model is generating over retrieved evidence rather than over its own memory. And, just as importantly, it makes the output checkable. A claim with a citation can be verified by a human or a second system in seconds. A claim without one is a assertion you have to take on faith, and faith is not a control. In a high-stakes process, an uncited answer should be treated as no answer at all.

Abstention and calibration: the model must be allowed to say "I don't know"

The most dangerous hallucination is not the wrong answer. It is the wrong answer delivered with total confidence, because confidence is what makes a human downstream stop checking.

This is why calibration matters more than raw accuracy in consequential work. A calibrated system is one whose expressed confidence tracks its actual reliability: when it says it is unsure, it genuinely is more likely to be wrong. Once you have that signal, you can do the thing an uncalibrated system cannot. You can let the model abstain. Below a confidence threshold, the correct output is not a guess. It is "I cannot determine this from the available evidence," which routes the case to a human instead of manufacturing a false certainty.

An agent that abstains well is more valuable in a regulated process than an agent that is slightly more accurate but never admits doubt, because the abstaining agent fails safely and the confident one fails silently. Optimizing purely for accuracy, and never for calibration, is how teams build systems that are right most of the time and catastrophic the rest of it.

Concordance evals: measure agreement on the decisions that matter

Standard accuracy metrics average over everything, which hides the failures you most need to see. A system can score well overall and still be unreliable on the small set of high-consequence, ambiguous cases where a wrong call does real damage.

Concordance evaluation targets that directly. Instead of asking "how often is the model right on average?", it asks "how often does the model agree with a trusted reference, or with a panel of expert humans, on the specific decisions that carry weight?" You measure agreement on the hard cases, not the easy ones, and you track it continuously rather than once at go-live. When concordance on a decision class drops, whether because the model was updated or because the input distribution shifted, you find out from the eval, not from an incident. This is the measurement that tells you whether the system still deserves to touch a given decision at all.

The REQUIRE-HUMAN gate: the agent prepares, the human decides

Grounding, abstention, and concordance reduce error and make it visible. They do not, and cannot, drive it to zero. So for the decisions where being wrong is unacceptable, the last control is not a better probability. It is a boundary.

A mature agent architecture resolves consequential decisions to one of a few explicit verdicts, and the one that matters here is REQUIRE-HUMAN. When a decision carries regulatory weight or irreversible consequence, the policy does not let the agent decide, no matter how confident it is. The agent does the work it is genuinely good at: it gathers the evidence, grounds and cites every relevant fact, checks the case against policy, and assembles a complete, reviewable package. Then it stops. A human makes the actual call, on top of everything the agent prepared.

This is not the agent failing to be autonomous. It is the correct division of labor. The agent compresses hours of assembly and verification into seconds and hands a person a decision that is fully worked and fully sourced. The person supplies the accountability that a probabilistic system cannot hold. The efficiency is real, and the consequential judgment stays where the law and good sense require it.

Why the law already agrees with this

There is a reason this architecture is not merely prudent. For a large class of decisions, it is close to a legal requirement.

GDPR Article 22 gives individuals the right not to be subject to a decision based solely on automated processing where that decision produces legal or similarly significant effects. A credit refusal, an insurance decline, a fraud block on someone's account: these are exactly the decisions the article is about. An architecture that lets an agent decide them alone is not just risking a confabulated call reaching a customer. It is standing on the wrong side of a right that already exists. The REQUIRE-HUMAN gate is not an optional governance nicety layered on for comfort. It is, for these decisions, how you stay lawful, and the newer regime around high-risk AI systems only sharpens the same expectation of meaningful human oversight.

So the reframe to carry into your next model discussion is this. The upgrade that makes your high-stakes agent safe is not a better model. It is grounding that makes answers checkable, calibration that lets the system abstain, concordance evals that watch the decisions that matter, and a human gate on the ones the law will not let a machine make alone. The model will keep improving, and that is welcome. It will never be the control. The governance is the control.

If you want to see which of your decisions belong behind a REQUIRE-HUMAN gate and which an agent can safely prepare, book a meeting.