Risks & limitations
Hallucination
Definition
A hallucination is model output that is fluent and confident but factually wrong — invented citations, non-existent functions, fabricated statistics. It stems from models being trained to produce plausible text, not verified truth.
Hallucination is not a bug to be patched but a consequence of the objective. A model trained to predict likely continuations will produce a likely-looking answer whether or not it knows the truth, and it has no internal signal distinguishing recall from generation.
Certain outputs are especially prone: specific numbers, citations, URLs, API signatures, legal and medical specifics, and anything after the knowledge cutoff. Confidence in tone carries no information about accuracy.
Mitigations that work: ground answers in retrieved sources, instruct the model to say when it does not know, require citations, and verify anything consequential against a primary source. Mitigations that do not work: asking the model whether it is sure.
Example
Asked for academic references on a niche topic, a model may produce perfectly formatted citations with plausible authors and journals that do not exist.
Related terms
RAG (Retrieval-Augmented Generation)
RAG retrieves relevant passages from your own documents and inserts them into the prompt before the model answers. It grounds responses in your data, cuts hallucination, and needs no retraining.
Grounding
Grounding means tying model output to verifiable sources — retrieved documents, search results, or database records — rather than relying on parametric memory. Grounded answers can be checked.
Knowledge cutoff
A model's knowledge cutoff is the date after which it has no training data. It cannot know about events, releases or prices after that point unless given the information in the prompt or via search.
Confidence score
A confidence score estimates how reliable an output is. Language models do not natively produce calibrated confidence, and asking a model how sure it is yields poorly calibrated self-assessment.
Prompt injection
Prompt injection is an attack where malicious instructions hidden in content the model processes override the developer's intended behaviour. It is the most serious unsolved security issue in LLM applications.
Lost in the middle
Lost in the middle is the tendency of models to use information at the start and end of a long context well while under-weighting material in the middle. Position affects whether the model actually uses what you provided.
Put this into practice
Understanding the term is step one. Our free courses and tools let you actually use it.