Retrieval-augmented generation tutorial

Steps

1. 1

   Understand the RAG architecture before building

   Retrieval-augmented generation combines two components: a retrieval system that finds relevant document passages for a given query, and a language model that generates an answer using those passages as context. The retrieval step is what prevents fabrication — the model is instructed to answer only from the provided passages, not from general training knowledge. Before writing any code, confirm that your use case is appropriate for RAG: it works well for question answering over a fixed document corpus, and less well for tasks that require synthesizing across the entire corpus or drawing on knowledge not present in the documents.

2. 2

   Prepare and chunk your documents

   Language models have context window limits, so you cannot pass an entire document corpus as context for every query. Documents must be split into chunks small enough to fit in context alongside other passages and the model's response. Chunk at natural boundaries — paragraphs, sections, or logical units — rather than at fixed character counts, which can split sentences mid-thought. Typical chunk sizes range from 256 to 1024 tokens. Smaller chunks retrieve more precisely but may lose context; larger chunks retain context but reduce precision. Test different sizes on a representative set of queries to find the right balance for your content.

3. 3

   Generate embeddings and build a vector index

   Retrieval in RAG works by comparing the semantic similarity between the query and document chunks. Both query and chunks are converted to embedding vectors — numerical representations of meaning — and the closest chunks are retrieved. Use an embedding model to convert each chunk to a vector, then store the vectors in a vector index or database that supports similarity search. The embedding model you use for documents and the one you use for queries must be the same model, because different models produce incompatible vector spaces. Rebuild the index whenever documents are added or updated.

4. 4

   Implement the retrieval step

   At query time, embed the user's question using the same embedding model used for indexing, query the vector index for the top-k most similar chunks (k=3 to 5 is a common starting point), and retrieve the corresponding chunk text. Test the retrieval step in isolation before connecting it to the language model: given a set of test questions, are the retrieved chunks actually relevant to those questions? Poor retrieval quality is the most common source of RAG failures, and diagnosing it requires inspecting retrieved chunks, not just final answers.

5. 5

   Construct the prompt and generate an answer

   Pass the retrieved chunks as context to the language model with instructions to answer based only on the provided information. A basic prompt structure: a system instruction that specifies the model's role and instructs it not to answer from general knowledge, the retrieved chunks clearly labeled as source material, and the user's question. Test that the model respects the grounding instruction: on questions where the answer is not in the retrieved chunks, the model should say it does not have enough information rather than generating a plausible-sounding answer from training data.

6. 6

   Evaluate and iterate on retrieval quality

   Evaluate the pipeline on a set of test questions with known correct answers from the corpus. For each question, check: were the correct chunks retrieved? Did the model answer accurately from those chunks? When the answer is wrong, trace back to whether retrieval or generation failed — retrieval failure is more common and is fixed by adjusting chunk size, the number of chunks retrieved, or the chunking strategy. Generation failures — where the right chunks were retrieved but the model answered incorrectly — point to prompt issues or model capability limits for the task.