Risks & limitations
Lost in the middle
Definition
Lost in the middle is the tendency of models to use information at the start and end of a long context well while under-weighting material in the middle. Position affects whether the model actually uses what you provided.
Measured across models and tasks, accuracy on retrieving a fact from a long context follows a U-shape: high when the fact is near the beginning or end, noticeably lower in the middle.
The practical consequence is that a large context window does not guarantee the model uses all of it. Filling 200,000 tokens does not mean 200,000 tokens of effective attention.
Design around it. Put critical instructions at the very start or very end. Order retrieved passages by relevance rather than document order. And prefer retrieving the relevant 5% over pasting everything and hoping.
Related terms
Context window
The context window is the maximum number of tokens a model can consider at once — your prompt, any attached documents, the conversation history, and the response it generates. Exceed it and the earliest content gets dropped.
Attention
Attention is the mechanism that lets a model weigh how much each token should influence the interpretation of every other token. It is how a model resolves what a pronoun refers to, or which clause modifies which noun.
RAG (Retrieval-Augmented Generation)
RAG retrieves relevant passages from your own documents and inserts them into the prompt before the model answers. It grounds responses in your data, cuts hallucination, and needs no retraining.
Chunking
Chunking splits documents into smaller passages before embedding them for retrieval. Chunk size is a key quality lever: too large dilutes relevance, too small loses the context needed to make sense.
Hallucination
A hallucination is model output that is fluent and confident but factually wrong — invented citations, non-existent functions, fabricated statistics. It stems from models being trained to produce plausible text, not verified truth.
Prompt injection
Prompt injection is an attack where malicious instructions hidden in content the model processes override the developer's intended behaviour. It is the most serious unsolved security issue in LLM applications.
Put this into practice
Understanding the term is step one. Our free courses and tools let you actually use it.