How Hallucination works in practice
Language models do not store facts as discrete, lookup-verified database rows; they compress statistical associations across billions of weights during training. When prompted to complete a sequence, the model samples tokens that maximise conditional likelihood. If a user asks about a niche UK statute, an unindexed internal policy, or a specific citation, the model bridges gaps in its weights by blending high-probability patterns into a plausible fabrication.
Practitioners distinguish between two failure modes: intrinsic hallucinations, where the model contradicts or misstates facts explicitly present in the supplied prompt or RAG context; and extrinsic hallucinations, where the model invents unverifiable details, URLs, case law citations, or software package names that do not exist in the context or the real world. High sampling temperature, ambiguous prompts, and noisy or truncated retrieval chunks all increase hallucination rates.
A well-known cautionary example occurred when litigants before the High Court and Manchester County Court submitted AI-generated skeleton arguments containing fabricated case citations that looked entirely authentic in style and neutral citation format, prompting formal judicial guidance from the Courts and Tribunals Judiciary.
Why it matters and trade-offs
In creative brainstorming or rough copy drafting, minor hallucinations are harmless friction. In regulated UK industries such as financial advice, healthcare triage, legal practice, and customer-facing contractual chatbots, a single hallucinated figure or fabricated policy promise creates regulatory liability, reputational damage, and binding financial loss (as demonstrated when airlines and retailers were held liable for discounts invented by support bots).
Hallucinations cannot be eliminated solely by buying a larger base model. Engineering teams control them through defence in depth: grounding answers via hybrid RAG, forcing inline source citations linked to chunk IDs, lowering temperature, instructing the model to abstain when evidence is absent, and running automated faithfulness evals before responses reach users.