Retrieval & data
RAG (Retrieval-Augmented Generation)
Definition
RAG retrieves relevant passages from your own documents and inserts them into the prompt before the model answers. It grounds responses in your data, cuts hallucination, and needs no retraining.
A RAG pipeline has two phases. Offline: split documents into chunks, embed each chunk as a vector, store them in a vector database. At query time: embed the question, find the most similar chunks, insert them into the prompt, and ask the model to answer using only that context.
RAG is the standard answer to "make the AI know about our company." It is far cheaper and more maintainable than fine-tuning, updates instantly when documents change, and can cite its sources — which matters enormously for trust.
The single most important line in any RAG prompt is the instruction to answer only from the provided context and say so when the answer is absent. Without it, the model fills gaps with plausible invention.
Example
Prompt template: "Answer using ONLY the context below. If the context does not contain the answer, say 'Not found in the provided documents.' Context: {retrieved chunks} Question: {user question}"
Related terms
Embedding
An embedding is a list of numbers representing the meaning of a piece of text, such that semantically similar texts have mathematically similar vectors. Embeddings make meaning-based search possible.
Vector database
A vector database stores embeddings and finds the most similar ones to a query vector quickly. It is the retrieval layer of most RAG systems. Examples include Pinecone, Weaviate, Qdrant and pgvector.
Chunking
Chunking splits documents into smaller passages before embedding them for retrieval. Chunk size is a key quality lever: too large dilutes relevance, too small loses the context needed to make sense.
Hallucination
A hallucination is model output that is fluent and confident but factually wrong — invented citations, non-existent functions, fabricated statistics. It stems from models being trained to produce plausible text, not verified truth.
Semantic search
Semantic search finds results by meaning rather than exact keywords, using embeddings to compare concepts. It matches "how do I get my money back" to a document titled "Refund Policy".
Reranking
Reranking takes an initial set of retrieved candidates and reorders them with a more accurate but slower model. It is one of the cheapest ways to materially improve RAG quality.
Put this into practice
Understanding the term is step one. Our free courses and tools let you actually use it.