Retrieval-Augmented Generation (RAG)

A production RAG pipeline has two distinct stages: indexing and runtime retrieval. During indexing, source documents such as PDFs, Confluence pages, and SQL records are split into semantically coherent chunks, typically 300 to 800 tokens long. Each chunk is passed through an embedding model to produce a dense vector and stored in a vector database alongside metadata like document ID, access-control tags, and publication date.

When a user asks a question, the system converts the query into a vector and runs a hybrid search combining dense vector similarity with lexical BM25 matching, enforced against the user's permission filters. A cross-encoder reranker then scores the top twenty candidates and selects the five most relevant passages. These passages are injected into the model's context window with strict system instructions to answer solely from the supplied evidence and cite each source.

For example, a London commercial law firm uses RAG across 400,000 executed contracts and Practical Law updates. When a solicitor asks how force majeure clauses were negotiated across retail leases in 2025, the pipeline retrieves only clauses the solicitor's practice group is authorised to view, returning a cited comparison in four seconds.

RAG solves the three hardest operational problems with off-the-shelf language models: knowledge cut-offs, unverifiable claims, and data access control. Compared with fine-tuning, which bakes static patterns into model weights at high compute cost and cannot easily revoke a deleted record, RAG updates the moment a document is re-indexed and enforces row-level security at query time. For most enterprise knowledge assistants, customer support bots, and policy search tools, RAG is the default starting architecture.

The trade-offs are retrieval latency, token cost, and pipeline fragility. Bad chunking splits tables or legal definitions in half, causing the retriever to miss critical context even when the answer exists in the corpus. Teams should not rely on naive vector-only RAG for multi-hop questions across hundreds of documents or whole-codebase reasoning, where GraphRAG, agentic search loops, or long-context models with prompt caching perform better.