Hallucination Follows from the Mechanism
A language model has no fact store to consult. From an enormous amount of writing it has learned statistically how text continues, and it picks the next token and then the next until the answer looks finished. No step in that loop checks the output against a source. The model is therefore not failing when it hallucinates, because right and invented answers come about the same way. Where the training data clearly supports a continuation, the model writes a fact, and where it does not, the model writes something shaped like a fact.
You therefore cannot buy a model that does not hallucinate. Every mitigation below either narrows the model’s task or checks its output from outside, and none changes the mechanism. A vendor who implies otherwise is selling a lower error rate as a new category.
Why It Sounds So Certain
The model errs in the same voice it uses when it is right, and that surprises users more than the error itself. It writes the invented clause without hesitating and without hedging.
A model does not measure its confidence; it has learned to write in a confident register. “I don’t know” is rarely the most probable continuation of a question that looks answerable, and the training objective rewards the text that fits, not calibrated doubt. The model therefore writes fluently and firmly by default, and that tone does not depend on whether the content is true.
Asking “are you sure?” therefore checks nothing. The model answers by the same process over nearly the same context. It re-derives the same error, or it gives in and agrees with your objection, whoever was right. Confidence scores the model reports about itself share the flaw. If you let the model judge its own work, you get another paragraph and no check.
When It Matters — and When It Doesn’t
Vendors are glad to answer the question “How accurate is the model?”. Two other variables determine the damage, though: what a wrong answer costs and how long it takes before somebody notices. Accuracy belongs to the model. Cost and time to detection belong to the process you put it in, and you design that process.
A machine builder runs a spare-parts assistant at its service desk. The model invents a part number, someone orders the wrong seal, the seal does not fit, and a technician reports it. That costs a few hundred euros and some annoyance, but the error shows itself. At a regional insurer, claims handlers have an assistant summarise cover before they write to a customer. The model omits an exclusion or invents one, and a denial letter goes out. Nobody notices until an Ombudsmann complaint arrives months later and someone has to explain the pattern across every letter sent since. Model and error rate are the same in both cases; the engineering task differs.
Spread the effort evenly across all four quadrants and you build a heavy approval workflow for an internal brainstorming tool and none for the letter to the customer. Sort your use cases into this grid before you buy anything; it shows where the money belongs.
What Reduces LLM Hallucinations
Three mitigations do most of the work. They appear here in the order teams usually try them, not in order of effect.
Grounding and its side effect
Retrieving your own documents and having the model answer from them helps most; how a RAG pipeline works is covered in a separate article. Grounding does more than lower the rate of hallucinations, though. It also changes their shape, and the new shape is harder to spot.
You can often recognise an ungrounded fabrication, because it stays vague, cites nothing and reads a little too smooth. A grounded one names a document, a section number and a quotation that is almost right, a real clause from a real policy that does not say what the sentence claims. Citations create trust before anyone checks them. A reviewer who would have doubted a bare assertion waves the same assertion through once “[Policy 4.2]” follows it. The engineering effort therefore belongs not in better-looking citations but in citations that sit one click from the source passage and highlight the quoted span, so that checking takes two seconds instead of two minutes. If checking costs more than trusting, people trust.
Constrain what it is allowed to say
Teams underrate this mitigation most, and it usually costs least. The reliable way to prevent invented part numbers is to make them impossible. If the answer must be a row in your ERP, the model should select the row and not write the number itself. Constrained decoding (a JSON schema, a GBNF grammar with llama.cpp, or Outlines) forces structurally valid output; a foreign-key check against the table forces the value to exist. Whatever fails either check never reaches a human.
If a hallucinated field fails the schema, the system catches it and retries. A hallucinated sentence in free prose, by contrast, comes back three weeks later as a support ticket, if one comes at all. Every degree of freedom you take from the output removes a category of hallucination you no longer have to detect afterwards.
Verify against something that is not the model
Verifying means holding the claim against an external source of truth: running the query, doing the arithmetic in code, confirming the ID resolves and diffing the summary against the passage it came from. An agent with tools is good for exactly this work, because it has to look things up rather than recall them. A second model as checker is better than nothing and catches some classes of error. It remains only weakly independent, though, because it shares training data, and with them blind spots, with the model it grades.
Verification catches invention far better than omission. A summary that silently drops the one exclusion that mattered is still fully supported by the source; it contradicts nothing and passes every entailment check you can build. I know of no clean technical fix for that, and it is the main reason a person still has to sit in the top-right quadrant.
What I Would Not Do
I would not make “hallucination rate” a KPI. It compresses everything into one number over a synthetic test set, it moves whenever someone touches the eval, and it does not say whether the remaining errors are the expensive ones. A cost-weighted score over a gold set of real queries, meaning your tickets and the clauses your handlers ask about, takes more effort and is worth more.
I would not buy a hallucination-detection model and present it as a safety concept. Such detectors flag part of a problem that cannot be fully detected, and the cases they miss do the damage. They work as a filter, not as a control.
I would also not put a model on a path that is both costly and slow to surface without a human who has the authority and the time to say no. If your system falls under the EU AI Act as high-risk — insurance pricing and risk assessment for life and health cover sits in Annex III — Article 14 requires meaningful human oversight; after the deferral by the Digital Omnibus, these obligations apply to Annex III systems from 2 December 2027. A claims handler expected to approve hundreds of summaries an hour rubber-stamps and does not oversee, and oversight you do not staff does not happen. Hallucination is also not always an accident. Whoever gets text into your context window can steer what the model confidently asserts, and prompt injection then becomes the same problem.
From our own build: a repair is not a proof
We ran into such a case ourselves while building the validation layer for Sentinel rules. A model wrote a detection rule that computed a rate per second from a time difference. The difference came in ticks, though, ten million per second, and the rate came out correspondingly too small. Microsoft’s parser accepted the rule, it ran without error, and it would never have reported an incident.
Shown the error, the model corrected the unit and in doing so swapped the arguments of the difference. The result could now never turn positive, and the rule would again never have fired. This version passed the parser too, and only the third held.
A model repairs what you show it and may break something else beside it; we work on that assumption. A second model would have found both versions plausible. A deterministic check that knows these patterns caught both. The validation layer page shows the complete run and the three error patterns.