RAG from Beginner to Expert: Retrieval Architectures for 2026

Retrieval-Augmented Generation (RAG) is the technique of giving a Large Language Model (LLM) access to knowledge it was never trained on — your PDFs, your company wiki, your database — by retrieving relevant pieces at question time and injecting them into the prompt. It is the backbone of almost every "chat with your documents" product.

This guide walks the full path: why retrieval exists, the classic pipeline, every major retrieval technique from TF-IDF to visual embeddings, the 2026 architecture landscape (agentic, vectorless, graph, cache-augmented), and the production concerns — evaluation, reranking, caching — that separate a demo from a system. No prior retrieval knowledge assumed; by the end you should be able to design, build, and critique a retrieval system.

Part 1: Why RAG Exists

An LLM has two hard limitations:

1. Knowledge cutoff — it only knows what was in its training data. Your internal documents, yesterday's policy update, your customer's contract: not in there.
2. Context window — you can paste documents into the prompt, but the window is finite (200K–1M tokens today), and every token you send costs money on every single request.

Three naive fixes, and why they fail:

RAG's answer: store the documents outside the model, find the few relevant pieces per question, send only those.

The single most important sentence in this guide: RAG quality is retrieval quality. If the right passage never reaches the prompt, no model — however smart — can answer correctly. Everything below is about getting the right passage into the prompt.

Part 2: Retrieval Foundations — Before Embeddings

Modern RAG stands on 50 years of information retrieval. Skipping this part is why many RAG systems underperform a 1990s search engine.

TF-IDF: Term Frequency × Inverse Document Frequency

The founding idea: a word matters for a document when it appears often in that document (TF) but rarely across all documents (IDF).

• TF — "cat" appears 3 times in a 100-word document → TF = 0.03
• IDF = log(N / df) — 1,000 documents total, "cat" in 10 of them → IDF = log(100) = 2. A stopword like "the" appears in ~990 → IDF ≈ 0.004, so it contributes nothing.
• Score = TF × IDF, summed over query terms, documents ranked by score.

BM25: TF-IDF, Fixed

BM25 (Best Matching 25) is the default lexical ranker in Elasticsearch, Lucene, and Solr. It fixes TF-IDF's two big flaws:

1. Term-frequency saturation — in TF-IDF, 100 mentions score 100× one mention. BM25 caps this: after the first several mentions, extra repetitions add almost nothing. Keyword stuffing stops winning.
2. Document-length normalization — a 300-word focused article beats a 10,000-word manual that mentions your term in passing. BM25 penalizes long documents relative to the corpus average length.

Two tuning knobs: k1 (saturation speed, typically 1.2–2.0) and b (length-penalty strength, typically 0.75).

Why lexical retrieval still matters in 2026: exact matches. Product codes, error strings, legal clause numbers, names — TAXCRM-4742 will never be found by "semantic similarity," but grep and BM25 find it instantly.

The Lexical Gap

What BM25 cannot do: understand that "car" and "automobile" are the same thing, or that "how do I get my money back" is asking about the refund policy. Vocabulary mismatch between the question and the document is the failure mode embeddings were invented to solve.

Part 3: Embeddings and Vector Search

What an Embedding Is

An embedding is a list of numbers (a vector) representing the meaning of a piece of text. An embedding model maps text into a space where semantically similar texts land close together:

"Can I work from home?"        → [0.23, -1.54, 0.87, ...]
"Remote work is allowed 3 days" → [0.25, -1.49, 0.90, ...]   ← close!
"Bring your pet on Fridays"     → [-1.10, 0.44, -0.32, ...]  ← far

Embedding dimension is the vector's length — 384 (MiniLM), 768 (BERT-base), 1024–3072 (commercial APIs). Higher dimension = more expressive but more storage and slower search. 384–1024 is the practical sweet spot for most systems.

Similarity Search

Retrieval becomes geometry: embed the question, embed every chunk (once, at indexing time), find the chunks whose vectors are closest — usually by cosine similarity (angle between vectors, 1.0 = identical direction).

At scale you never compare against every vector. ANN (Approximate Nearest Neighbor) indexes — HNSW being the dominant algorithm — find near-closest vectors in milliseconds over millions of entries. This is the core service a vector database provides: pgvector (inside Postgres), Qdrant, Chroma, Weaviate, Milvus, Pinecone.

One Misconception, Killed Early

You cannot send an embedding to an LLM instead of text. LLM APIs accept tokens (text) only. Embedding your question does not compress it or reduce its token cost — the question still travels to the LLM as full text. The saving in RAG comes from a different place entirely: retrieval selects a few small chunks instead of sending the whole corpus. (Embeddings do enable one genuine cost trick — semantic caching, covered in Part 8.)

Part 4: The Classic RAG Pipeline

4.1 Parsing

The unglamorous step that decides everything downstream. A PDF is not text — it is positioned glyphs. Tables, multi-column layouts, headers/footers, and scanned pages all break naive extractors. Tools in rough order of power: PyMuPDF / unpdf (fast, plain text), unstructured (layout-aware), Docling (tables, reading order), and vision-model parsing for the hardest documents. Garbage in, garbage retrieved.

4.2 Chunking

Embedding models and retrieval both work best on focused passages, so documents get split into chunks. The method matters:

Rules of thumb: 200–800 tokens per chunk; store metadata with each chunk (source file, page numbers, section title) — you need it for citations; and prefer structure-aware splitting whenever the document has structure.

4.3 Retrieval and Generation

Embed the question → top-k similarity search (k typically 3–10) → assemble the prompt:

System: Answer ONLY from the provided sources. If the sources don't
contain the answer, say so. Cite the source for every claim.

Sources:
[1] employee-handbook.pdf, p.12: "Remote work is permitted up to 3 days..."
[2] it-policy.pdf, p.4: "Remote workers must use company-approved..."

Question: Can I work from home, and what equipment do I need?

The grounding instruction ("answer only from sources") plus citations is what turns an LLM from a plausible-sounding guesser into an auditable answering system.

4.4 Small-to-Big Retrieval

A quiet upgrade with a big payoff: search small, return big. Embed small chunks (precise matching) but hand the LLM the parent section the chunk came from (full context). Also called parent-document retrieval. Most frameworks support it natively; hierarchical systems (Part 6) get it for free.

Part 5: Making Retrieval Actually Good

The classic pipeline above is a baseline. These four techniques are where real systems gain their quality — roughly in order of return on effort.

5.1 Hybrid Search

Run both BM25 and vector search, merge the results. Lexical catches exact terms (codes, names); semantic catches paraphrases. The standard merge is Reciprocal Rank Fusion (RRF) — score-agnostic, no normalization headaches:

RRF(doc) = Σ over each ranker: 1 / (k + rank_in_that_ranker)    (k ≈ 60)

A document ranked #1 lexically and #3 semantically beats one ranked #2 in just one list. Elasticsearch, Qdrant, and Weaviate ship hybrid + RRF as a built-in.

5.2 Reranking

First-stage retrieval (BM25, vectors) is built for speed over millions of chunks and is deliberately crude. A reranker is a second model that takes the top 50–100 candidates and re-scores each (question, chunk) pair with full cross-attention — far more accurate than comparing two independently-computed vectors.

Options: Cohere Rerank, Voyage rerank, open-source bge-reranker. Cost: one cheap model call, tens of milliseconds. This is usually the single highest-leverage addition to a RAG system.

5.3 Query Transformation

The user's question is often a bad search query. Fix the query before retrieving:

• HyDE (Hypothetical Document Embeddings) — ask an LLM to imagine an answer first, embed that hypothetical answer, and search with it. A fake answer lives in the same semantic space as real answers, so document-to-document matching beats question-to-document matching. Best for short, vague queries; costs one extra LLM call and can mislead if the hypothetical is wrong.
• Multi-query expansion — generate 3–5 rephrasings, retrieve for all, merge with RRF.
• Query decomposition — split "Compare the 2024 and 2026 VAT thresholds and their effect on small businesses" into sub-questions, retrieve per sub-question, answer from the union.
• Step-back prompting — first ask the more general question ("what are the VAT registration rules?"), retrieve for that, then answer the specific one.

5.4 Metadata Filtering

Attach structured fields to every chunk (department, date, document type, tenant) and filter before similarity search. "What changed in the 2026 policy?" should never even look at 2019 documents. Trivial to implement, routinely forgotten, massive precision win in multi-tenant or time-sensitive corpora.

Part 6: The 2026 Architecture Landscape

Everything so far assumed the classic embed-and-search shape. The last two years produced serious alternatives — each with a legitimate niche. This is the map:

6.1 Long Context — No Retrieval At All

Context windows reached 1M tokens. For a bounded, stable set of documents (roughly up to 20), skip retrieval entirely: pass the documents in the prompt, enable prompt caching so repeat requests reuse the processed prefix at ~10% cost. Claude additionally accepts PDFs natively and can return page-level citations. Zero infrastructure, full cross-document reasoning. Fails on: large corpora (cost), frequently-changing content (cache invalidation), and precision (long-context models can lose needles in very long prompts).

6.2 CAG — Cache-Augmented Generation

The self-hosted cousin of long context: preload the whole corpus into the model's KV cache once, persist that cache to disk, restore it before every query. Zero retrieval step, minimal per-query latency. Wins when the corpus is small, stable, shared across users, and latency-critical. Loses the moment documents change often or the corpus outgrows the window.

6.3 Vectorless RAG — Reasoning-Based Retrieval

Popularized by PageIndex. No embeddings, no vector DB. At ingest, build a table-of-contents tree per document: root (doc summary) → chapters → sections, each node holding a summary; leaves hold the actual text. At query time, an LLM reads the outline and reasons about where the answer lives — like a human using a book's table of contents.

Strengths: no vector infrastructure (a relational DB holds the tree), exact page-level citations for free, preserves document structure, strong on structured documents (laws, manuals, financial reports — PageIndex hit 98.7% on FinanceBench). Costs: 2–3 LLM calls per question instead of one cheap embedding lookup, and recall depends entirely on summary quality — a fact not reflected in any section summary is invisible to the navigator.

6.4 GraphRAG — Knowledge-Graph Retrieval

Extract entities and relationships from documents into a graph, cluster into hierarchical communities, summarize each community. Retrieval traverses the graph. Uniquely good at multi-hop, cross-document relationship questions — "which suppliers are connected to the delayed project through subcontractors?" — where chunk similarity fundamentally cannot help, because the answer is an edge path, not a passage. Cost warning: graph extraction is LLM-heavy (Microsoft's original implementation famously cost ~$33K to index a large corpus; 2026 variants like LazyGraphRAG defer extraction to query time and cut this dramatically). Reach for it only when your question distribution is genuinely relationship-shaped.

6.5 Visual Retrieval — ColPali / ColQwen

Skip text extraction entirely: embed each PDF page as an image using a vision-language model. Each page becomes a grid of patch embeddings; queries match via late interaction (MaxSim, the ColBERT idea applied to vision). No OCR, no parsing, no chunking — and it sees tables, charts, stamps, handwriting, and layout. On visually dense financial PDFs, visual retrieval hits ~84% recall where text-only pipelines manage ~62%. This is also the only approach on this list that handles scanned documents natively. Costs: heavy embeddings (~1030 vectors per page — around 1.3GB per 10K pages), GPU for the vision encoder, and the answering model must also accept images.

6.6 Grep-Based Agentic Search — No Index At All

The Claude Code approach: documents live as plain files; an agent uses grep/glob/read tools in a loop — search, read, refine, repeat. No index to build, maintain, or invalidate; results always reflect the current bytes on disk. Amazon Science (2026) measured keyword-driven agentic search at >90% of RAG performance with no vector database. The trade: many tool-call round-trips burn tokens and latency per query. Great when an agent harness already exists and the corpus is text-friendly; poor fit for high-QPS user-facing chat.

6.7 Agentic RAG — The Umbrella

The 2026 consensus architecture for hard questions: retrieval becomes a tool, and the LLM drives a loop — search, read, decide it's not enough, reformulate, search again, then answer. Any retriever from this list (vector, hybrid, tree navigator, graph, grep) can be the tool inside the loop. Vectorless RAG is agentic retrieval with a fixed two-step loop; grep search is agentic retrieval with lexical tools. The price of the loop is latency and tokens; the payoff is multi-hop questions and self-correction when the first retrieval misses.

6.8 What Fine-Tuning Is (and Isn't) For

Fine-tuning (LoRA/QLoRA) changes a model's behavior — tone, format, domain vocabulary — not reliably its knowledge, and baked-in knowledge goes stale on the first document update. It is a complement, not an alternative: RAFT (Retrieval-Augmented Fine-Tuning) trains the model to use retrieved context faithfully and ignore distractor passages. The production pattern: tune the interface, retrieve the content — roughly 60% of serious deployments combine both.

Part 7: Evaluation — The Most-Skipped, Most-Important Part

Without measurement, every tuning decision — chunk size, HyDE on/off, reranker choice — is guesswork. Evaluate the two stages separately:

Retrieval metrics (does the right chunk get retrieved?):

Generation metrics (given the chunks, is the answer good?):

• Faithfulness / groundedness — is every claim supported by the retrieved text? (The anti-hallucination metric.)
• Answer relevance — does it actually address the question?
• Citation accuracy — do the citations point at the right sources?

Practical recipe: build a test set of 50–200 (question → gold answer → gold source) triples from real user questions; run retrieval metrics on every pipeline change; use an LLM-as-judge (or the RAGAS framework) for the generation metrics; and only then A/B test in production. The order matters — evaluate retrieval first, because a generation-stage fix cannot rescue a retrieval-stage miss.

Part 8: Production Concerns

Semantic caching. Embed each incoming question; if a previously-answered question sits above a similarity threshold, return the stored answer and skip the LLM entirely. For FAQ-shaped traffic this eliminates a large share of calls. (This is the one legitimate way embedding a question saves money.)

Prompt caching. Providers cache the processed prompt prefix — put stable content (system prompt, document context) first, volatile content (the question) last, and repeat requests pay ~10% for the cached portion. This is what makes the long-context/no-RAG option economically viable.

Index freshness. Documents change. Re-embed changed chunks only (hash pages or sections to detect changes); avoid full re-indexes. Vectorless/tree systems re-ingest per document, which is a natural incremental unit.

Multi-tenancy and security. Retrieval must respect permissions — filter by tenant/ACL in the retrieval query, never post-hoc in the prompt. A leaked chunk in the context window is a data breach; the model cannot be trusted to "not look at" text it was given.

Cost model. Classic RAG per query: one embedding call (negligible) + one LLM call over ~2–6K context tokens. Agentic/vectorless: 2–10 LLM calls. Long-context: one call over the whole corpus (cache-discounted). Measure per-answer cost — architectures differ by an order of magnitude.

Failure UX. When retrieval finds nothing relevant, say so — a grounded "I don't have that information" beats a fluent hallucination. Design the fallback message path as a first-class feature, and log every fallback: those logs are your corpus-gap report.

Part 9: The Decision Table

These compose. A serious 2026 system often looks like: metadata filter → hybrid retrieval (BM25 + vectors) → reranker → LLM with citations, wrapped in an agentic loop for hard queries, with semantic caching in front and an evaluation harness underneath.

Part 10: The Learning Path, Recapped

1. Foundations — TF-IDF → BM25; embeddings, dimensions, cosine similarity; build a toy ChromaDB/pgvector search.
2. Classic pipeline — parsing, chunking strategies, top-k retrieval, grounded prompting with citations.
3. Quality stack — hybrid + RRF, reranking, HyDE and friends, metadata filtering, small-to-big.
4. Architectures — long context/CAG, vectorless, GraphRAG, visual, grep-agentic, agentic loops; know the decision table cold.
5. Engineering maturity — evaluation harness first, then caching, freshness, tenancy, cost, failure UX.

The field moves fast — "RAG is dead" headlines appear yearly — but the underlying skill is durable: understanding how to connect a question to the right evidence, at acceptable cost, with verifiable output. Retrieval was a hard problem before LLMs and remains one after them; the architectures in this guide are just today's answers to it.