RAG Architecture That Actually Works in Production

Every RAG tutorial follows the same script: split documents, embed them, stuff them into a prompt, get an answer. It works in notebooks. It falls apart in production.

Emanuel Mallia has spent the past two years helping teams deploy RAG systems that serve real users with real consequences. The gap between the tutorial version and the production version is enormous, not because the concept is wrong, but because the tutorials skip everything that matters. This is part of the AI delivery series, and RAG is where most teams first discover that gap.

Here's what actually works.

The Architecture That Ships#

Production RAG is not "retrieve and generate." It's a pipeline with at least six stages, each doing real work:

Each stage exists because we learned the hard way what happens without it:

• Query rewriting. Users ask bad questions. "What's the policy on that thing from last week?" needs to become a specific, searchable query. LLM-based query expansion or decomposition handles this.
• Hybrid search. Semantic search alone misses exact terms. BM25 alone misses intent. You need both.
• Re-ranking. Your initial retrieval casts a wide net. A cross-encoder re-ranker sorts the results by actual relevance, not just embedding similarity.
• Context compression. You retrieved 20 chunks but your context window shouldn't contain all of them verbatim. Extract the relevant passages.
• Citation validation. If your system can't point to the source, it's not enterprise-ready. Every claim needs a traceable reference.

Skip any of these stages and you'll hit a failure mode that looks like a model problem but is actually a pipeline problem.

Chunking Is the Foundation#

I've seen teams spend weeks comparing embedding models while using naive fixed-size chunking. This is backwards. Chunking strategy has more impact on retrieval quality than model choice.

Recursive chunking as the default#

Start with recursive character splitting. It respects document structure (paragraphs, then sentences, then words). Fixed-size chunks with overlap sound reasonable but they split mid-thought constantly.

The parent-child pattern#

This is the single highest-impact pattern I've deployed. Small chunks for precise retrieval, parent chunks for complete context.

interface ChunkNode {
  id: string;
  content: string;
  parentId: string | null;
  children: string[];
  metadata: {
    source: string;
    page: number;
    section: string;
    level: 'parent' | 'child';
  };
}

function buildChunkHierarchy(
  document: string,
  parentSize: number = 1500,
  childSize: number = 300
): ChunkNode[] {
  const nodes: ChunkNode[] = [];

  // Split into large parent chunks first
  const parents = recursiveSplit(document, parentSize);

  for (const parent of parents) {
    const parentNode: ChunkNode = {
      id: generateId(),
      content: parent.text,
      parentId: null,
      children: [],
      metadata: {
        source: parent.source,
        page: parent.page,
        section: parent.section,
        level: 'parent',
      },
    };

    // Split each parent into smaller child chunks
    const children = recursiveSplit(parent.text, childSize);

    for (const child of children) {
      const childNode: ChunkNode = {
        id: generateId(),
        content: child.text,
        parentId: parentNode.id,
        children: [],
        metadata: {
          source: parent.source,
          page: parent.page,
          section: parent.section,
          level: 'child',
        },
      };
      parentNode.children.push(childNode.id);
      nodes.push(childNode);
    }

    nodes.push(parentNode);
  }

  return nodes;
}

// At retrieval time:
// 1. Search against child chunks (precise matching)
// 2. Return parent chunks (complete context)

You embed and search over the child chunks. When a child matches, you return its parent to the LLM. The model gets complete context. Retrieval gets precision. Everyone wins.

Document-aware parsing#

PDFs with tables, multi-column layouts, and headers need structure-aware parsing. Don't treat a table as a sequence of text lines. Tools like Unstructured or custom parsers that preserve table structure and section hierarchy are worth the setup cost. I've seen retrieval accuracy jump 30-40% just from fixing the parsing layer.

Retrieval That Works#

Hybrid search is non-negotiable#

Pure semantic search fails on exact terms: product codes, policy numbers, technical identifiers. Pure keyword search fails on conceptual queries. Every production system I've built uses both.

The fusion strategy matters. Reciprocal Rank Fusion (RRF) is the practical default:

• Run BM25 and semantic search independently
• Assign each result a score based on its rank position: 1 / (k + rank)
• Sum scores across both result sets
• Sort by combined score

RRF is simple, requires no training, and works surprisingly well. A typical k value of 60 is a reasonable starting point.

Re-ranking is the highest-ROI addition#

If you do one thing beyond basic RAG, add a cross-encoder re-ranker. The improvement is immediate and measurable.

Initial retrieval (BM25 + semantic) gives you recall, a broad set of potentially relevant chunks. The re-ranker gives you precision: the actually relevant ones, properly ordered. Models like Cohere Rerank or open-source cross-encoders handle this well.

Don't: retrieve 5 chunks and pass them all to the LLM. Do: retrieve 20-30 chunks, re-rank them, pass the top 5.

When to use HyDE vs multi-query#

HyDE (Hypothetical Document Embeddings): the LLM generates a hypothetical answer, you embed that answer, and search for similar real documents. Useful when user queries are very different from document language, such as customer questions against technical documentation.

Multi-query: the LLM generates 3-5 reformulations of the original query, you run each independently, and merge results. Better for ambiguous queries where the user might mean several things.

Both add latency. Use them selectively, not as defaults.

The Evaluation Gap#

Most teams build RAG systems without measuring retrieval quality. They evaluate vibes. "The answers look good" is not a metric.

Minimum viable metrics#

You need four numbers, measured continuously:

• Faithfulness. Does the response only contain information from the retrieved context? Hallucination detection.
• Answer relevance. Does the response actually answer the question?
• Context precision. Are the retrieved chunks relevant to the question?
• Context recall. Did retrieval find all the relevant information?

The RAGAS framework automates these measurements using LLM-as-judge evaluation. It's not perfect, but it's dramatically better than manual spot-checking.

Golden test sets#

Build a set of 50-100 question-answer pairs with known correct answers and the source documents they come from. Run every pipeline change against this set. This catches regressions that you'd otherwise only discover from user complaints.

CI/CD gates#

Treat retrieval quality like test coverage. Set thresholds:

• Faithfulness below 0.85? Block the deploy.
• Context precision dropped by more than 5%? Investigate before merging.

This sounds heavy. It saves you from shipping broken retrieval to production and spending a week debugging user complaints.

Enterprise Reality#

Three things that tutorials never mention but production demands:

PII handling. Documents contain personal data. Your chunks inherit that data. You need PII detection in the ingestion pipeline and clear policies on what gets embedded. Retrieval results must respect data classification.

Access control at the chunk level. User A can see document X but not document Y. Your retrieval layer needs to enforce this, not just at the document level, but at the chunk level, because a single document might contain sections with different access requirements. This means every chunk carries permission metadata, and every query filters by the requesting user's access rights.

Audit trails. When a user gets an answer, you need to know which chunks were retrieved, which were passed to the LLM, and what the model generated. This is not optional for regulated industries. The same data observability discipline that applies to pipelines applies here; log the full pipeline state for every request.

When NOT to Use RAG#

Not everything needs a vector database. I've seen teams build full RAG pipelines for problems that have simpler solutions.

Small, stable corpora. If your knowledge base is under 200 pages and changes infrequently, consider cache-augmented generation: preload the entire corpus into a long-context model. No chunking, no retrieval, no pipeline complexity. Modern models handle 100K+ tokens well.

Structured data questions. "What were our sales last quarter?" doesn't need RAG. It needs text-to-SQL or a structured API. Choosing the right database for your workload matters more than the retrieval layer in these cases. Building a RAG system over database exports is a common antipattern.

Simple lookups. If the answer is always in one specific place (a lookup table, a configuration file, a single API endpoint), don't over-engineer it. A direct integration is simpler and more reliable.

The right question isn't "how do we build RAG?" It's "does this problem actually require retrieval over unstructured text?"

Frequently Asked Questions#

What is the best chunking strategy for RAG systems?#

The parent-child chunking pattern delivers the highest impact in production RAG systems. Small child chunks (around 300 tokens) are used for precise retrieval, while their larger parent chunks (around 1,500 tokens) provide complete context to the LLM. This gives you retrieval precision without sacrificing the context the model needs to generate accurate answers.

Why should RAG systems use hybrid search instead of pure vector search?#

Pure semantic search fails on exact terms like product codes, policy numbers, and technical identifiers, while pure keyword search fails on conceptual queries. Production RAG systems should combine BM25 keyword search with vector search using Reciprocal Rank Fusion (RRF) to merge results. Adding a cross-encoder re-ranker on top is the single highest-ROI improvement you can make.

How do you evaluate RAG system quality in production?#

Production RAG evaluation requires four metrics measured continuously: faithfulness (does the response only use retrieved context), answer relevance (does it answer the question), context precision (are retrieved chunks relevant), and context recall (did retrieval find all relevant information). Build a golden test set of 50-100 question-answer pairs and gate deployments on quality thresholds.

Honest Assessment#

RAG is mature enough for production. The patterns are well-understood, the tooling has improved significantly, and the failure modes are documented. But it requires the same engineering discipline that production AI agents demand: observability, guardrails, and iteration limits.

The retrieval is the hard part, not the generation. Most teams over-invest in prompt engineering and under-invest in chunking, indexing, and evaluation. Flip that ratio.

Build the evaluation framework first. Measure before you optimise. Test your pipeline like you'd test any other critical system. The teams that ship reliable RAG systems are the ones that treat retrieval as an engineering problem, not a model problem.