Hallucinations

What is an AI hallucination?

Hallucinations are a major blocker for putting LLMs into regulated, high-stakes enterprise workflows, and the single biggest reason to combine LLMs with Retrieval-Augmented Generation (RAG) and evaluation.

An LLM generates text one token at a time by predicting what comes next given the prompt and its training. When the model doesn't know an answer, or was never trained on it, it doesn't stop; it produces the most probable-sounding continuation. That continuation can be wrong, fabricated, or contradictory to source material. The output reads correct, which is what makes it dangerous.

The term "hallucination" comes from the fact that the model asserts things confidently that aren't there. In practice enterprise teams care about two questions: how often does this happen on our workload, and what does it cost us when it does. Both are measurable.

Why do LLMs hallucinate?

• Training-time gaps. The model was never exposed to the fact or was exposed to conflicting versions of it.
• Frozen knowledge. The model's world stops at its training cutoff.
• Compression loss. Facts stored in weights are lossy summaries of the training text.
• Prompt ambiguity. A vague prompt gets a vague, sometimes invented, answer.
• Long context degradation. In long inputs, models can lose track of what was actually said versus what they "know."
• Sycophancy. Models trained with RLHF can prefer answers users like over answers that are true.

Types of hallucination

Type

Definition

Typical fix

Intrinsic

Contradicts the source you gave it (RAG, doc, spreadsheet)

Better retrieval, groundedness checks

Extrinsic

Unsupported by any source in the prompt or ground truth

RAG, tool use, refuse-when-uncertain prompts

Fabricated citation

Invented URL, paper, court case, product feature

Verify all cited entities against a real index

Reasoning error

Correct facts, wrong inference

Chain-of-thought, self-verification, tool use for calculations

How to reduce hallucinations

The pattern that works is "defense in depth": layered mitigations so no single failure produces a wrong answer to the user.

• RAG with citations. Answer only from retrieved passages; require the model to cite them. See the Enterprise h2oGPTe RAG docs.
• Grounded generation. Prompt engineering patterns that explicitly say "if the answer isn't in the passages, say you don't know."
• Guardrails. Structured output enforcement, allow/deny lists, safety classifiers, prompt injection defenses.
• Function calling for facts. Route factual queries (dates, math, database lookups) to real tools instead of the model's memory.
• Fine-tuning for style, not facts. Teach the model how to speak in your domain; keep the facts in RAG. Fine-tune with the H2O LLM Studio docs.
• Evaluation on real traffic. Score groundedness continuously with H2O Eval Studio; see LLM evaluation.
• Human-in-the-loop. For high-stakes actions, require review before the answer becomes an action.

How to measure hallucinations

• Groundedness score. For a RAG answer, what fraction of the claims are supported by the retrieved passages?
• Factuality score. For a general answer, does it agree with a trusted knowledge base?
• Faithfulness. Does the answer accurately reflect the input?
• Human eval. SME rating on a fixed sample, still the gold standard for high-risk tasks.

You cannot manage what you cannot measure. Every production LLM deployment should score hallucinations on a fixed evaluation set, at every model or prompt change.

Where hallucinations matter most

• Healthcare, life sciences, finance, legal, government: a wrong answer has legal or safety consequences.
• Customer-facing assistants: brand damage from wrong claims.
• Code generation: hallucinated APIs waste hours of engineering time.
• Compliance and audit: a fabricated regulation cite is worse than "I don't know."

AI hallucinations in H2O AI

Enterprise h2oGPTe combines RAG with source citations, groundedness checks, and configurable guardrails so answers are supported by real documents. H2O Eval Studio measures hallucination rates on your own data, including groundedness, faithfulness, and safety. For the highest-stakes deployments (banks, hospitals, federal agencies), both run on-prem or air-gapped, so measurement and mitigation happen on infrastructure the customer controls.