Hallucination (AI)

An AI hallucination is when a generative AI model produces output that sounds confident and plausible but is factually incorrect, fabricated, internally inconsistent, or unsupported by the source material it was given. The term applies most often to LLMs inventing fake citations, wrong statistics, or non-existent entities — but covers any case where the model "makes things up."

Hallucinations are a structural feature of how LLMs work, not a bug to be patched. Models predict the most plausible next token given context; when grounded knowledge runs out, they confidently extrapolate. A 2025 Stanford evaluation found frontier models hallucinated on 27% of factual questions without retrieval, dropping to under 4% when paired with RAG — proof that hallucination is contextual, not innate to the model.

Why hallucinations happen

Several mechanisms produce hallucinations:

• Knowledge gaps — The model wasn't trained on the relevant fact and confidently invents one to fill the gap.
• Spurious pattern matching — The model produces output that fits the style of correct answers but not the substance (fake URLs, fake citations in a credible format).
• Conflicting training data — Contradictory facts in the training corpus cause the model to average toward the wrong answer.
• Long-context degradation — Models perform worse deep in long contexts; relevant information gets lost.
• Sycophancy — The model agrees with incorrect user assertions to seem helpful.
• Decoding randomness — Higher sampling temperature produces more creative but more error-prone outputs.

A 2026 Anthropic interpretability paper showed that some hallucinations correlate with specific neurons activating "confident speculation" features — and that these can be partially suppressed via activation steering, but not eliminated.

Mitigations

Production AI systems use a combination of mitigations:

• RAG — Ground generation in retrieved sources. Most effective single mitigation.
• Structured output — Force the model to emit JSON conforming to a schema; reduces freelance fabrication.
• Tool use — Let the model call calculators, databases, or APIs instead of guessing facts.
• Self-verification — Have the model critique and revise its own output, ideally with a different prompt.
• Confidence calibration — Trained models that say "I don't know" when uncertain (not yet reliable).
• Eval suites — Continuous testing against known-answer benchmarks; regressions trigger rollback.
• Human review — High-stakes domains (medical, legal) keep humans in the loop.

No single mitigation eliminates hallucination. Defense in depth is required.

Examples of notable hallucinations

1. Lawyers fined ($5,000) for filing a brief with fake case citations generated by ChatGPT (Mata v. Avianca, 2023).
2. Air Canada chatbot fabricated a refund policy; the airline was held legally bound to honor it (2024 ruling).
3. Google's Bard demo (Feb 2023) incorrectly stated the JWST took the first exoplanet image; Alphabet stock dropped $100B that day.
4. CNET retracted dozens of AI-written articles after readers identified factual errors and math mistakes.
5. Glue-on-pizza incident — Google's AI Overviews cited a Reddit joke as legitimate cooking advice (May 2024).

How PostKit relates to hallucinations

For PostKit, hallucination is a constrained problem because the task is generation, not factual retrieval. PostKit isn't asking models "what's true?" — it's asking them "what's a compelling caption for this brand?" — a task where creativity is welcome and there's no single correct answer.

But hallucination still matters in three ways:

1. Brand fact accuracy — If a brand profile says "we serve 12 cities," PostKit must not let the model invent additional cities. Mitigation: facts from the profile go in structured fields, never in free-text prompts where the model could embellish.
2. Statistic citations — If users want PostKit to include statistics in posts, those stats must be real. PostKit's roadmap includes a citation-verification step that retrieves and validates any claim before publishing.
3. Format compliance — A "format hallucination" (returning JSON with extra fields, wrong structure) is a real failure. PostKit uses structured output schemas (JSON Schema, Pydantic), retry-on-validation-failure, and post-processing to enforce format.

The discipline mirrors broader AI engineering: assume the model can hallucinate any unstructured output, and design constraints that make hallucination either impossible or detectable.

Frequently asked questions

Are hallucinations getting better as models improve? Yes, but slowly. GPT-5's hallucination rate is roughly half of GPT-4's on factual benchmarks, but the floor is non-zero. Architecture-level improvements (RAG, tool use) close the gap faster than model improvements alone.

Why does the model sound so confident when it's wrong? LLMs are trained to produce coherent, fluent text. Confidence is a stylistic feature, not a calibration of truth. Without explicit "uncertainty" training, models default to the confident voice of their training corpus.

Can I tell when a model is hallucinating? Sometimes. Red flags: overly specific claims without sources, unfamiliar entities, contradictions within the same response, citations that don't resolve. Tools like Honest AI and Anthropic's "I don't know" training help.

Do all generative models hallucinate? Yes. Image models invent objects (extra fingers, impossible architecture). Video models invent physics. Audio models invent words. The phenomenon is universal across generative AI.

Is hallucination just a fancy word for "wrong"? Some critics argue yes, that "hallucination" anthropomorphizes statistical errors. The term is industry-standard, but "confabulation" or "fabrication" are more technically accurate.

Does fine-tuning reduce hallucinations? Sometimes. Fine-tuning on accurate domain data can reduce hallucinations in that domain; fine-tuning on noisy data amplifies them. Fine-tuning to say "I don't know" can backfire by making the model overly conservative.

What's the legal liability for AI hallucinations? Emerging area. Air Canada (2024) was held liable for chatbot statements. The general rule: the deployer is responsible for outputs, regardless of model provider. Disclaimers help but don't fully insulate.

Related terms

• Generative AI
• LLM (Large Language Model)
• RAG (Retrieval-Augmented Generation)
• Prompt engineering
• Fine-tuning
• AI agent

Sources

• Stanford HAI — Foundation Model Transparency Index (2026)
• Mata v. Avianca, Inc. — US District Court (2023)
• Anthropic — Mapping the Mind of a Large Language Model (2024)
• OpenAI — GPT-5 System Card hallucination evaluations (2025)