RAG Architectures: The Complete Builder's Guide to Retrieval-Augmented Generation (2026)

The model gets the blame for a bad answer from a RAG (Retrieval-Augmented Generation) system. Often the search deserves it.

RAG is a way to make an LLM (Large Language Model) answer from your documents instead of from memory. The system first searches a knowledge base for the passages most relevant to the question, then gives those passages to the model together with the question and asks it to answer using them. The pattern was named in a 2020 paper by Patrick Lewis and colleagues, and it has since become a standard way to connect a model to private, fresh or very large bodies of knowledge.

If you only remember one line from this guide, make it this: a RAG system is a search system with a language model attached to the end, and the model can only use what the search hands it. Treat it as only a prompt problem and you risk tuning the wrong half.

Five questions builders ask (quick answers)

• What is RAG, in one line? Search first, then generate. The model answers from retrieved passages and can cite them.
• What are the stages of a RAG pipeline? Eight: parse, chunk, enrich, embed, store, retrieve, rerank, generate. Each has its own failure modes, and seven of the eight happen before the answering model sees a single passage.
• Which RAG architecture should I start with, and when do I move to agentic? Start with "advanced RAG" (hybrid search, filters, reranking) for production document Q&A (question and answer) systems. Move to agentic or graph RAG only when an evaluation shows a failure those fix.
• When is RAG the wrong tool? When the knowledge base is small and stable (put it in the prompt), when the question is a count or a total (use a database), and when the problem is style or format (use prompting or fine-tuning).
• What should I do first? Write 40 real questions (a starting minimum) with the page that answers each one. That file anchors every later choice.

A scene you will recognise

This is an illustrative scenario, not a real company. Picture an Indian company with 3,000 employees and an HR (Human Resources) policy library spread across PDF (Portable Document Format) files, a wiki and old email attachments. The HR team spends its days answering the same questions: how many days of leave carry over, what the relocation policy says for a transfer from Pune to Hyderabad, whether the new maternity benefit applies to contract staff.

Someone builds a chatbot over the documents in a weekend. The demo is great. In week two an employee asks about notice periods and the bot quotes a clause from a policy that was replaced last year. In week three a manager asks something that only appears in a table inside a scanned PDF, and the bot says confidently that no such policy exists.

Nobody changed the model. The failures sit in the search half of the system: a stale document in the index, a table the parser never read. That is the whole argument for treating RAG as an engineering discipline rather than a prompt.

Three habits sink RAG projects: tuning prompts before checking the search, shipping without a set of real test questions, and enforcing permissions in the prompt instead of in retrieval. The sections below show how to avoid each.

Why RAG exists

A language model knows what it saw in training, frozen at a cutoff date, and it does not know your private documents. It also tends to produce fluent answers when it is unsure. RAG addresses four problems at once:

• Private knowledge: your contracts, tickets, policies and code were never in the training data.
• Freshness: update the index and the answers change, with no retraining.
• Grounding: the model is told to answer from supplied passages, which cuts down on invented facts when the instructions and checks are good.
• Traceability: every answer can point to the passage it came from, which is what reviewers, auditors and sceptical users want.

It does not remove hallucination. A model can still misread a passage, ignore it or blend it with something it half remembers. That is why the later sections spend as much time on measurement as on construction.

How a RAG system works: two pipelines

Every RAG system is two pipelines sharing one index.

• The indexing pipeline (offline): runs when documents are added or changed. It reads documents, splits them into pieces called chunks, attaches information about each piece, converts each chunk into a list of numbers called an embedding, and stores everything in a searchable index.
• The query pipeline (online): runs on every question. It finds candidate chunks, orders them, builds a prompt from the best ones, and asks the model to answer with citations.

An embedding is a list of numbers that represents the meaning of a piece of text, produced by an embedding model. Texts with similar meaning get embeddings that sit close together, so "How many casual leaves do I get?" lands near a chunk about leave entitlement even when the words differ. Searching by that closeness is called vector search or dense retrieval.

The eight stages

Notice that seven of the eight stages happen before the answering model sees anything. When an answer is wrong, check retrieval first, because the generator can only work with what it was given.

The architecture ladder: naive to agentic, with graph as a side branch

"RAG architecture" means how these stages are arranged and how much decision-making you give the system. Think of a four-rung ladder with one side branch. Each rung solves a failure of the one below and costs more in latency, money and complexity. The side branch, Graph RAG, changes how the index is built instead. For a wide catalogue of named variants, see 26 Types of RAG Systems. Here is the practical ladder.

Rung 1: Naive RAG

Chunk the documents, embed them, retrieve the top few by vector similarity, paste them into the prompt. It takes an afternoon and is the right way to learn and to demo. It breaks on exact terms (vector search is weak at matching a policy number or a function name), returns near-duplicate chunks, and has no way to say that one source outranks another.

Rung 2: Advanced RAG

Add the things that fix rung 1's known failures: hybrid retrieval (keyword and vector together), metadata filters, query rewriting, a reranker, and a prompt that asks for citations and allows "I could not find this". This is the sensible default for production question answering over documents. Before reaching for something more exotic, check whether rung 2 done carefully already solves the problem.

Rung 3: Modular RAG

The pipeline becomes a set of swappable components with routing. A router looks at the question and decides which index, tool or pipeline to use: policy documents for one question, a database query for another, a general answer with no retrieval for a third. Add fallbacks and caches. Use this rung when you serve several corpora or several kinds of question and one pipeline cannot be good at all of them.

Rung 4: Agentic RAG

The model itself decides whether to retrieve, what to search for, whether the results are good enough, and whether to search again with a different query or a different tool. This handles multi-step questions such as "compare the leave policy for contract staff in 2023 and today". The price is more model calls, more latency, less predictable cost, and behaviour that is harder to test. It needs a proper harness (the tools, limits, checks and memory around the model) and hard limits on the loop (see Loop Engineering): a cap on steps, a cap on spend and a clear stopping rule.

Side branch: Graph RAG

Instead of (or as well as) a pile of text chunks, build a knowledge graph: entities such as people, products and suppliers, and the relations between them, extracted from the documents. Retrieval can then follow relationships, and summaries of clusters of related entities can answer broad questions like "what are the main themes across these 10,000 reports". Use it when answers depend on connections across many documents. Expect heavy indexing cost and a graph that needs maintenance as documents change. Graph RAG is a different axis from the rungs above: it changes what the index holds, not how much the system decides, so it can sit under a simple pipeline or an agentic one.

Variants you will meet on the way

• Self-RAG trains a model to judge its own retrieval and output as it generates. CRAG (Corrective RAG) adds a retrieval evaluator that triggers a fallback, such as a web search, when the retrieved passages look poor. Both are examples of grading retrieval and retrying.
• Multimodal RAG: retrieves over images, charts, tables and audio as well as text, either by embedding them directly or by generating text descriptions and indexing those.
• Long-context hybrids: retrieve a few whole documents or sections rather than small chunks, and let a large context window do the reading.

The rule for climbing: start at advanced RAG for production, and move to another rung or add the graph branch only when your evaluation shows a specific failure it fixes. "Agentic sounds more advanced" is not a failure.

Is RAG the right tool? A fitment guide

One costly mistake is building RAG where it was never needed. Check the problem against this table before choosing an architecture.

Many real systems combine these. A support assistant might use RAG for policy text, a database tool for order status and a router to pick between them. This is also where context engineering comes in: RAG is one of several ways to decide what the model sees.

Fitment by use case

The decisions inside each stage

Each of these deserves its own deep dive, and several will get one in this series. Here is what to decide and what to avoid.

1. Parse: look at the text your system actually sees

Parsing is easy to under-check. PDFs hide tables, columns, headers and footers that scramble reading order, and scanned pages need OCR (Optical Character Recognition). Open the parsed output for a sample of your real documents and read it as the model would. Keep headings, page numbers and table structure, remove repeated headers and footers, and record which version of each document you ingested.

2. Chunk: keep an answer whole

A chunk should hold a complete thought. A reasonable starting hypothesis (not a proven rule) is chunks about three times the length of a typical answer, with overlap about as long as one answer, then testing against your own questions. Prefer splitting along real structure (sections, clauses, headings) over blind fixed-size cuts. Chunking has enough depth for a guide of its own, so this section keeps to the essentials. One idea worth carrying over: store small chunks for matching but hand the model the larger surrounding section (often called parent-child or small-to-big retrieval).

3. Enrich: metadata and context

Attach to every chunk: source file, page, section heading, document date, version, language, owner and, critically, who is allowed to see it. These fields power filters, citations and freshness rules. Mark superseded versions (for example with an end date) so they can be excluded from retrieval, and keep the access field in step with the source system: when someone's permissions change, the index must change too.

A second kind of enrichment fixes a common retrieval miss: a chunk that reads "The deadline is 30 days" means nothing on its own. In Anthropic's Contextual Retrieval technique, a model writes a short explanation of where each chunk sits in its document, and that text is added to the chunk before it is embedded and before it is keyword-indexed. In Anthropic's tests the share of relevant chunks that failed to appear in the top 20 results fell from 5.7% to 3.7% with contextual embeddings, to 2.9% when contextual keyword search was added, and to 1.9% when reranking was added as well. Their reranking step scored 150 candidates and kept 20, which is more than the pipeline sketched later in this guide, so do not expect that exact figure from a smaller setup. These are Anthropic's results on their own test sets, so measure on yours before assuming the same gain. The cost is one model call per chunk at indexing time.

4. Embed: choose on your data, not on a leaderboard

Public rankings such as MTEB (Massive Text Embedding Benchmark) are a good place to build a shortlist. They do not tell you how a model handles your jargon, your languages or your document lengths. Run your questions through two or three candidates and compare recall. With only 40 questions, a gap of a few points is noise, so grow the set before you treat a small difference as a decision. Rules that save pain later:

• Use the same embedding model for documents and queries, in its query mode for questions if it has one (some models expect a different prefix for queries).
• Record the model name and version with every stored vector. Changing the model means re-embedding everything.
• Larger vectors cost more to store and search. Check whether a smaller size loses anything on your questions before paying for the bigger one, and check your database's limits: for example, pgvector can index standard vectors up to 2,000 dimensions and its half-precision type up to 4,000.
• If your content mixes languages, test with real mixed queries, not clean ones.

5. Store: pick for your scale and your filters

You need a database that can do ANN (approximate nearest-neighbour) search, which finds close vectors quickly without comparing against every one. The two index families you will meet:

• HNSW (Hierarchical Navigable Small World): a layered graph. Fast queries and high recall, but it uses a lot of memory and builds slowly.
• IVF (Inverted File index): groups vectors into clusters and searches only the nearest clusters. Lighter on memory than HNSW, especially when combined with compression, but it needs tuning and recall depends on how many clusters you search. For small collections, an exact scan with no index is often fast enough.

Settle these strategy questions before you commit to a store:

• Memory and cost: holding vectors in memory can be the biggest bill. Compressing them (quantization) trades a little accuracy for much less memory, so measure the recall you lose.
• Multi-tenancy: one shared index with a tenant filter is cheap, but isolation is then one filter bug away from a leak. Separate collections or databases per tenant cost more and isolate better. Choose by how bad a leak would be.
• Updates and deletes: confirm the store can delete and re-index documents without a full rebuild, because stale content produces wrong answers.
• Hybrid support: check whether keyword and vector search can run in one query, or whether you will merge them yourself.
• Managed or self-hosted: managed services save operations work, while self-hosting gives control over where data lives, which matters if the documents hold personal data.

For choosing a store: if you already run PostgreSQL and your collection is modest, the pgvector extension keeps everything in one place and supports both index types. Dedicated vector databases such as Qdrant, Weaviate, Milvus and Pinecone, and search engines such as Elasticsearch and OpenSearch, earn their place at larger scale, with heavy filtering, or when you want built-in hybrid search and operational features. A detail that is easy to miss is filtered search: if the system finds the nearest 10 vectors first and applies the permission or date filter afterwards, you can end up with two results or none. This is a known behaviour of approximate indexes in pgvector, where filters apply after the index scan. Version 0.8 added iterative index scans that keep searching until enough rows pass the filter, and other stores handle filtering in their own ways. Switch the relevant option on, then test recall with your real filters and real user permissions.

6. Retrieve: go hybrid early, then fuse

Vector search finds meaning but is weak on exact strings. Keyword search, usually ranked with BM25 (Best Matching 25, a classic formula that rewards rare matching words), finds exact strings but misses paraphrases. Run both and merge. A common way to merge is RRF (Reciprocal Rank Fusion): each result earns 1 / (k + rank) from every list it appears in, and the scores are summed. A value of 60 for k is a common default. RRF ignores the raw scores, which is useful because keyword scores and vector similarities are on incompatible scales. The cost is that it also ignores how much better one result is than the next; weighted fusion of normalised scores is the main alternative.

Retrieve wide, then narrow: as a starting point, fetch 20 to 50 candidates, rerank, and keep only the best few. A few more techniques, each a response to a specific failure:

• Query rewriting or expansion: turn a vague or conversational question into one or several search queries. Fixes: users who ask in half-sentences.
• Query decomposition: split a compound question into parts and retrieve for each. Fixes: multi-part questions.
• HyDE (Hypothetical Document Embeddings): have the model write a pretend answer and search with that. Fixes: short questions whose wording is far from the documents'. Risk: the pretend answer can drift.

7. Rerank: sharpen the order

The first-stage retriever is built for speed across millions of chunks. A reranker is slower but sharper: a cross-encoder model reads the question and one candidate passage together and scores how well the passage answers the question. Run it only on a short list, 20 to 50 candidates as a starting range. Options include hosted reranking services and open-source cross-encoder models. It adds cost, latency and another component to run, so measure both what it costs and what it gains. It tends to pay off when the right chunk is retrieved but ranked too low, and does little when the right chunk is already first and the model misreads it.

Other ways to retrieve or order results, each solving a different problem:

• Late-interaction models (ColBERT is the best known) keep a vector for every word-piece and compare them at query time. They can be more accurate than one vector per chunk and are cheaper than a cross-encoder, at the price of much more storage.
• Model-based rerankers ask a language model to score or order the candidates. They can follow instructions such as "prefer the most recent policy", but cost more and run slower per query.
• MMR (Maximal Marginal Relevance) picks results that are relevant and also different from those already chosen, which cuts near-duplicate passages.
• Metadata boosts raise the score of newer or more authoritative documents. Treat them as tie-breakers only: a boost guarantees nothing, so superseded versions should be excluded by a filter or removed from the index, not merely ranked lower.

8. Generate: make grounding the rule

• Tell the model to answer only from the numbered sources, to cite them, and to say plainly when the sources do not contain the answer.
• Keep the context lean. More passages are not always better, and models can under-use information buried in the middle of a long context (the "Lost in the Middle" research by Nelson Liu and colleagues, 2023). Newer models vary, so test with yours. Put the strongest sources at the start or end of the context, where models use information best.
• Return the sources with the answer, and check citations in code: confirm the quoted text actually appears in the chunk cited.
• Where your model lets you set it, use a low temperature for factual answers.
• Treat retrieved text as untrusted data. A document can contain instructions aimed at the model, so keep passages clearly separated from your instructions and give the model only the tools it needs.

A sketch of an advanced RAG query pipeline

This is not tied to any library. It shows the shape, and the shape is what you need to explain to a coding agent or a reviewer.

def rrf(ranked_lists, k=60):
 scores = {}
 for results in ranked_lists: # each list holds chunk ids in rank order
 for rank, chunk_id in enumerate(results, start=1):
 scores[chunk_id] = scores.get(chunk_id, 0) + 1 / (k + rank)
 return sorted(scores.items(), key=lambda item: item[1], reverse=True) # (chunk_id, score)

def answer(question, history, user):
 allowed = permissions_for(user) # filter BEFORE ranking
 queries = rewrite(question, history) # uses the chat so far, for follow-ups
 lists = []
 for q in queries:
 lists.append(keyword_search(q, allowed, limit=50))
 lists.append(vector_search(embed_query(q), allowed, limit=50))
 fused = rrf(lists)[:50]
 chunks = load_chunks([cid for cid, _ in fused]) # text plus source, page, version
 top = rerank(question, chunks)[:6] # cross-encoder; each keeps its metadata and score
 top = [c for c in top if c.score >= MIN_RELEVANCE] # drop weak matches
 if not top:
 return "I could not find this in the policy documents.", []
 reply = generate(build_prompt(question, top)) # numbered sources, abstain rule
 reply = verify_citations(reply, top) # quoted text must appear in the cited chunk
 log_trace(question, queries, fused, top, reply)
 return reply, top # always return the sources

In the sketch, embed_query uses the embedding model's query mode, and the minimum-relevance check makes abstaining a property of the pipeline instead of a hope about the prompt. Set that threshold from your evaluation, not by guesswork.

Twelve practices that pay off

1. Start from questions, not tools. Collect at least 30 to 50 real questions with the answer and the source page for each, and keep adding from real user queries. This is your first evaluation set.
2. Evaluate retrieval and generation separately. Otherwise you cannot tell which half failed.
3. Go hybrid early. Keyword and vector search fail in opposite ways.
4. Try a reranker before upgrading the model. It often helps when the right chunk is retrieved but ranked low. Measure what it costs in latency.
5. Enforce permissions in retrieval, not in the prompt. A prompt that says "do not reveal confidential documents" is a request. A filter that never retrieves them is a control. Keep the permission data in the index in step with the source system, because a stale copy of someone's access lets the filter pass what it should block.
6. Make "I do not know" a feature. Test that the system abstains when the sources do not answer, and count it as a success.
7. Version everything. Embedding model, chunker settings, prompt and index version. Any change means re-running the evaluation.
8. Run the index like a product. Handle deletes and updates, show effective dates, and alert on documents that have not been refreshed.
9. Log the trace. Question, rewritten queries, candidate ids and scores, final context, answer and user feedback. Without it you cannot debug a bad answer. Keep personal data out of logs where you can.
10. Keep a person in the loop for high-stakes answers. HITL (Human-in-the-Loop) review belongs wherever a wrong answer has legal, financial or safety consequences.
11. Watch cost per answered question. Cache embeddings and repeated answers, limit candidate counts, and use a small model for query rewriting.
12. Climb the architecture ladder only on evidence. Every rung is more to build, test and pay for.

What not to do

• Do not ship on a demo. A handful of hand-picked questions proves nothing about the other thousand.
• Do not set the number of retrieved chunks to 3 and hope. Retrieve wide, rerank, then trim.
• Do not rely on one chunk size for every document type. Policies, tickets, contracts and code split differently.
• Do not change the embedding model without re-indexing. Mixed vectors give silent nonsense.
• Do not tune on your evaluation questions. Keep a held-out set you never look at while adjusting.
• Do not use a language model to judge itself without checking the judge. Compare its scores with human labels on a sample first.
• Do not use RAG to do a database's job. Totals and counts need a query, not a paragraph.
• Do not trust retrieved text. A document can carry instructions meant for the model. Treat passages as data, never as commands.

When something is wrong: symptom, cause, fix

How to measure a RAG system

Measure the two halves separately.

Retrieval quality asks whether the right passage was found and how high it ranked:

• Hit rate at k: in what share of questions does at least one right chunk appear in the top k results?
• Recall at k: of all the chunks that answer a question, what share appear in the top k? It equals hit rate when one chunk is enough.
• Precision at k: of the top k results, how many are relevant?
• MRR (Mean Reciprocal Rank): the average of 1 divided by the rank of the first correct result, so a first-place hit scores 1 and a fifth-place hit scores 0.2.
• nDCG (Normalized Discounted Cumulative Gain): rewards putting the most relevant results first when relevance comes in grades.

Answer quality asks whether the model used the passages well:

• Faithfulness (groundedness): is every claim in the answer supported by the retrieved passages? An answer faithful to a stale passage still scores well, so faithfulness is not correctness.
• Answer correctness: does the answer contain the key facts it should, checked against the expected answer for each test question?
• Answer relevance and completeness: does it answer the question that was asked, fully?
• Abstention accuracy: does it decline when the sources do not contain the answer, and answer when they do?

Open-source and commercial tools such as Ragas, TruLens, DeepEval and LangSmith can compute many of these. Whatever you use, calibrate any model-based judge against human labels on a sample, and keep the question set growing from real user queries. A full guide to writing evaluations for RAG and for agents is coming in this series.

Brief your coding agent properly

If you build with a coding agent, the quality of your brief decides the quality of the system. A good brief states the architecture, the constraints and, above all, how "done" is checked. Put standing rules in the project's instruction file (see our guide to Claude Code markdown files) and give the task something like this:

Build a question-answering service over our HR policy documents.

Stack: Python 3.12, FastAPI, Postgres with pgvector 0.8 or later.
Retrieval: keyword search (Postgres full-text search, or a BM25 extension)
plus vector search, fused with reciprocal rank fusion, then rerank the top
50 candidates down to the top 6.
Each chunk stores: source file, page, section heading, effective date,
document version, whether it is superseded, and the user groups allowed
to see it.
Filter by the caller's groups BEFORE ranking. Turn on pgvector's iterative
index scans and test that restricted users still get full result lists.
Answers: use only the numbered sources, cite them as [1], [2], and reply
"I could not find this in the policy documents" when they do not answer.
Send document text only to the embedding and reranking services named in
README.md. Use an embedding size that pgvector can index.

Ingestion first:
1. Parse the PDFs, with OCR for scanned pages, and write the parsed text of
 20 documents to parsed_samples/ for a person to read (tables included).
2. Record the embedding model name and version on every chunk.

Evaluation first:
3. A person supplies evals/golden.jsonl (about 40 real questions, each with
 the expected source page, the key facts the answer must contain, the
 asking user's groups, and whether the right answer is "not found") and
 evals/holdout.jsonl, which you never see. Never edit either file. If
 golden.jsonl does not exist yet, stop and ask for it. Do not invent
 questions.
4. Build `make eval`. It reports recall@50 on the candidate list, hit rate
 on the final 6, faithfulness (judged by a model whose scores a person
 has compared with hand-labelled answers), answer correctness against the
 key facts, and abstention accuracy.

Done means: `make eval` meets the thresholds in evals/thresholds.json (set
by a person from a baseline run, and never lowered by you), all tests pass,
and a trace is logged for every answer.

Three things in that brief matter more than the stack. A person writes the evaluation file, because an agent left to invent its own "real" questions will write easy ones that echo the documents and then mark its own homework. The brief names the checks that decide "done", with thresholds the agent cannot lower. And constraints such as "filter before ranking" are stated explicitly, because that is the kind of requirement an agent will otherwise quietly skip.

If you are building in India: what changes in your design

Where to go next

This guide is the map. Upcoming posts in the series go deep on chunking strategies, vector database choices, retrieval methods, ranking and reranking, and how to write evaluations that catch RAG failures before your users do.

Your next step: write the 40-question file described in the coding-agent brief above, with one row per question: the question, the page that answers it, the key facts the answer must contain, and whether the right answer is "not found". Run it against your current system before you change anything. That one run tells you whether you are stuck in search or stuck in generation.

One question to take back to your team this week: if a user asked about a policy that changed last month, could your system prove which version it answered from? 👇

Frequently asked questions

What is RAG in simple terms?

RAG, short for Retrieval-Augmented Generation, means searching a knowledge base for relevant passages and giving them to a language model along with the question, so the answer is based on those passages rather than the model's memory alone.

What is the difference between RAG and fine-tuning?

RAG supplies knowledge at question time, so updating the index updates the answers. Fine-tuning changes the model itself using examples and is better for consistent style, format or a narrow task. Use RAG when the problem is missing or changing knowledge, and fine-tuning when the problem is behaviour.

Do I still need RAG now that context windows are so large?

Often yes, but not always. A small, stable knowledge base can go directly into the prompt, and Anthropic's guidance in 2024 was that a knowledge base smaller than about 200,000 tokens could go straight into the prompt, though you should leave headroom and test accuracy at that length. Larger or frequently changing collections, per-user permission rules and cost control still favour retrieval, and long contexts can under-use information placed in the middle.

What is the best chunk size for RAG?

There is no universal best. A reasonable starting hypothesis, not a proven rule, is chunks about three times the length of a typical answer with overlap about as long as one answer, then test on your own questions. Splitting along real structure such as sections and clauses usually beats fixed-size cuts.

What is hybrid search and why does it matter?

Hybrid search runs keyword search and vector search together and merges the results, commonly with Reciprocal Rank Fusion. Vector search catches paraphrases, keyword search catches exact terms like policy numbers and error codes, and each fails where the other succeeds.

What does a reranker do?

A reranker re-scores the top candidates from the first search by reading the question and each passage together, which is more accurate than comparing precomputed vectors. It is slower, so it is applied to a short list, 20 to 50 candidates as a starting range. It tends to pay off when the right chunk is retrieved but ranked too low, and it adds latency and cost, so measure both.

When should I use agentic RAG?

Use it when questions need several searches and judgement, such as comparing documents or gathering evidence from different sources. It costs more, runs slower and is harder to test, so first confirm that a well-built advanced RAG pipeline is failing on those questions.

How do I reduce hallucinations in a RAG system?

Retrieve better, keep the context lean, instruct the model to answer only from numbered sources and to say when they do not answer, verify citations in code, and measure faithfulness on a test set. No single step removes hallucination, so measure rather than assume.

Which vector database should I choose?

If you already use PostgreSQL and your collection is modest, pgvector keeps things simple. Dedicated vector databases and search engines make sense at larger scale, with heavy filtering or when you want hybrid search built in. Test recall and speed at your real data size with your real filters turned on.

How do I know if my RAG system is any good?

Build a set of real questions with known answers and source pages. Measure retrieval (hit rate and recall at k, Mean Reciprocal Rank) and answer quality (faithfulness, correctness, correct abstention) separately, and re-run the set whenever you change the model, the chunking or the prompt.

Learn to build this with us

To follow this series as each post publishes, join the Garage Labs community. If you want to build these systems with live guidance, RAG, agent frameworks, harnesses, loops and evaluations are core topics in the Applied AI Practitioner Program, taught through live sessions and build labs. See the programme page for the schedule and details.

Sources and further reading

• Patrick Lewis and colleagues, Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (2020)
• Anthropic, Introducing Contextual Retrieval (19 September 2024)
• Nelson F. Liu and colleagues, Lost in the Middle: How Language Models Use Long Contexts (2023)
• Gordon Cormack, Charles Clarke and Stefan Buettcher, "Reciprocal Rank Fusion outperforms Condorcet and individual Rank Learning Methods" (SIGIR, the Special Interest Group on Information Retrieval conference, 2009)