Hallucination Is the Mechanism, Not a Malfunction
A language model has no fact store to consult. It holds a statistical model of how text continues, learned from an enormous amount of writing, and it uses that to choose the next token, then the next, until the answer looks finished. There is no step in that loop where it checks anything against anything. So the honest answer to why LLMs produce confident nonsense is an uncomfortable one: hallucinations are not the model failing. Right answers and invented ones come off the same production line by the same method. When the training data strongly supported a continuation, you get a fact. When it did not, you get something fact-shaped.
That distinction is not academic, because it tells you what you cannot buy. There is no model coming that has stopped hallucinating, any more than there is a hammer coming that has stopped being hard. Every mitigation below works either by narrowing what the model is asked to do, or by checking its output from the outside. None of them touch the mechanism. A vendor implying otherwise is selling you a lower error rate and calling it a different category of thing.
Why It Sounds So Certain
What catches people out is not that the model is wrong. It is that it is wrong in exactly the voice it uses when it is right. There is no tell, no hesitation, no thinner prose around the invented clause.
Confidence, in a model, is a register it learned to write in — not a measurement it performs. Hedging text is comparatively rare in the training corpus, and “I don’t know” is almost never the most probable continuation of a question that looks answerable. Nothing in the training objective ever rewarded calibrated doubt; it rewarded producing the text that fits. Fluent certainty is therefore the default style, and it is uncorrelated with whether the content is true.
The consequence most teams get wrong: asking the model “are you sure?” is theatre. The follow-up is generated by the same process over nearly the same context, so it produces a fluent re-derivation of the original error — or, worse, folds and agrees with your pushback regardless of who was right. Self-reported confidence scores share the flaw. If your verification step is the model’s opinion of its own work, you have added a paragraph, not a check.
When It Matters — and When It Doesn’t
“How accurate is the model?” is the wrong question, and it is the one every vendor is delighted to answer. Two other variables decide whether a hallucination actually hurts you: what a wrong answer costs, and how long it takes before somebody notices. Accuracy is a property of the model. Those two are properties of the process you drop it into, and they are the ones you control.
Take a machine builder running a spare-parts assistant for its service desk. The model invents a part number; someone orders the wrong seal; it arrives, it does not fit, and a technician says so. Annoying, a few hundred euros, and — crucially — self-announcing. Now take a regional insurer whose claims handlers use an assistant to summarise cover before writing to a customer. The model omits an exclusion, or invents one. A denial letter goes out. Nobody notices at all, until an Ombudsmann complaint lands months later and someone has to explain the pattern across every letter sent since. Same model, same error rate, entirely different engineering problem.
Most teams smear their effort evenly across all four boxes, which is how you end up with a heavyweight approval workflow on an internal brainstorming tool and nothing at all on the letter that goes to a customer. Sort your use cases into this grid before you buy anything. It will tell you where the money goes.
What Actually Reduces LLM Hallucinations
Three mitigations carry most of the weight. They are listed here in the order people usually try them, which is not the order of effectiveness.
Grounding — and the trap it sets
Retrieving your own documents and making the model answer from them is the biggest single lever, and how a RAG pipeline works is a topic in its own right. The part that gets glossed over: grounding does not only lower the rate of hallucinations, it changes their shape — and the new shape is harder to catch.
An ungrounded fabrication often looks like one: vague, unsourced, a bit too smooth. A grounded one arrives with a document name, a section number, and a quotation that is almost right — a real clause, retrieved from a real policy, that does not actually say what the sentence claims it says. Citations transfer trust before anyone verifies them. A reviewer who would have squinted at a bare assertion will wave the same assertion through once it has “[Policy 4.2]” after it. So the useful engineering is not prettier citations; it is citations that are one click from the source passage, with the quoted span highlighted, so that checking costs two seconds instead of two minutes. Make verification cheaper than trusting, or people will trust.
Constrain what it is allowed to say
The most underrated mitigation, and usually the cheapest. The reliable way to stop a system from inventing a part number is to make invented part numbers unrepresentable. If the answer must be one of the rows in your ERP, then writing the number is not the model’s job — selecting the row is. Constrained decoding (a JSON schema, or a GBNF grammar with llama.cpp, or Outlines) forces the output to be structurally valid; a foreign-key check against the actual table forces it to exist. Anything that fails either check never reaches a human.
The asymmetry is the point. A hallucinated field that fails a schema check is a caught error and a retry. A hallucinated sentence in free prose is a support ticket in three weeks, if you are lucky enough to get one. Every degree of freedom you remove from the output is an entire category of hallucination you no longer have to detect after the fact.
Verify against something that is not the model
Real verification means checking the claim against an external source of truth: run the query, do the arithmetic in code, confirm the ID resolves, diff the summary against the passage it came from. This is where a tool-using AI agent earns its keep — not by being clever, but by being made to look things up rather than recall them. A second model as a checker is better than nothing and genuinely catches some classes of error, but it is only weakly independent: it shares training data, and therefore shares blind spots, with the model it is grading.
And the honest limit: verification catches invention far better than it catches omission. A summary that quietly drops the one exclusion that mattered is fully supported by the source, contradicts nothing, and sails through every entailment check you can build. I do not know of a clean technical fix for that. It is the main reason the top-right quadrant still needs a person.
What I Would Not Do
I would not make “hallucination rate” a KPI. It collapses into a single number over a synthetic test set, it moves whenever someone touches the eval, and it says nothing about whether the errors that remain are the expensive ones. A cost-weighted score over a gold set built from real queries — the actual tickets, the actual clauses your handlers actually ask about — is harder to produce and worth more.
I would not buy a hallucination-detector model and call it a safety story. Detectors flag some percentage of a problem that is not fully detectable, and the number they miss is the number that hurts. They are a filter, not a control.
And I would not put a model on a path that is both high-cost and slow-to-detect without a human who has the authority and the time to say no. If your system falls under the EU AI Act as high-risk — insurance pricing and risk assessment for life and health cover sits in Annex III — Article 14 requires meaningful human oversight, and a claims handler expected to approve hundreds of summaries an hour is not oversight, they are a rubber stamp with a job title. Oversight you have not staffed is oversight you do not have. Worth remembering too that hallucination is not always accidental: if an attacker can get text into your context window, they can steer what the model confidently asserts, which is where prompt injection and LLM security stop being separate topics from this one.
Where Tippel Fits
Most of the work in making an LLM system trustworthy is not prompt engineering. It is deciding which quadrant each use case belongs in, then constraining, grounding, and checking in proportion — and being willing to say that one particular path should not have a model on it at all. That judgement is what the AI Readiness Check produces: your real data, one use case, a fixed window, and an honest verdict on whether the error modes are ones your process can absorb.
If you have an idea where a confident wrong answer would be expensive, that is worth an hour of conversation before it is worth a budget. Get in touch and we will work out which quadrant you are in.