Risks & limitations
Confidence score
Definition
A confidence score estimates how reliable an output is. Language models do not natively produce calibrated confidence, and asking a model how sure it is yields poorly calibrated self-assessment.
Models express confidence in tone regardless of accuracy, and self-reported confidence correlates weakly with correctness. "I'm 95% certain" from a model is not a probability.
More useful signals exist. Token log-probabilities give a rough measure of the model's own uncertainty over its output. Self-consistency — sampling several times and measuring agreement — is a reasonable proxy: high disagreement indicates low reliability. For RAG, retrieval scores indicate whether relevant context was even found.
Use these to route rather than to report. Escalate low-confidence cases to review instead of surfacing a false precision to users.
Related terms
Hallucination
A hallucination is model output that is fluent and confident but factually wrong — invented citations, non-existent functions, fabricated statistics. It stems from models being trained to produce plausible text, not verified truth.
Self-consistency
Self-consistency runs the same reasoning prompt several times at non-zero temperature and takes the most common answer. It trades cost for accuracy on problems with a single verifiable result.
Human in the loop
Human in the loop means inserting a person at decision points in an automated workflow — typically to approve consequential actions or review low-confidence outputs before they take effect.
Evaluation (evals)
Evaluation is systematically measuring AI output quality against a test set, rather than judging from a handful of examples. It is what makes prompt and model changes safe to ship.
Prompt injection
Prompt injection is an attack where malicious instructions hidden in content the model processes override the developer's intended behaviour. It is the most serious unsolved security issue in LLM applications.
Lost in the middle
Lost in the middle is the tendency of models to use information at the start and end of a long context well while under-weighting material in the middle. Position affects whether the model actually uses what you provided.
Put this into practice
Understanding the term is step one. Our free courses and tools let you actually use it.