RAG in production: chunking, retrieval, and hallucination control nobody documents

The demo worked perfectly. Two hundred product documentation pages, loaded into a vector store over a long afternoon, and the assistant was returning accurate answers with real citations. Recall was high enough that the team stopped measuring it. The product manager was delighted. The contract was signed.

Six weeks later, the customer pilot went live over 12,000 internal documents — policies, SOPs, historical case files, technical specs, a decade of institutional knowledge scanned and OCR'd into PDF. The first thing the customer tried was a question that should have been trivial: "What is the correct escalation procedure for a Tier-2 incident?" The answer came back with three citations. Two of them were wrong. One cited a document that mentioned the word "escalation" in a completely different context. The third was plausible but outdated — it cited the 2019 version of the procedure, not the 2023 revision that was also in the corpus.

The team investigated. The retrieval step was returning chunks with cosine similarity scores above 0.82 — which on the 200-document demo had been a reliable signal for relevance. On 12,000 documents, the same threshold was returning chunks that were topically adjacent but not actually answering the question. The language model, presented with those chunks, did what language models do: it synthesized a confident-sounding answer from what it had been given. The citation mechanism pointed at the chunks that had been retrieved, not at the chunks that had actually contributed to the answer — and because the retrieval was wrong, the citations were wrong too.

This failure mode is not exotic. It is the default outcome for RAG systems that move from demo to production without an explicit engineering investment in retrieval quality. The architecture that gets you from zero to a working prototype in a weekend — embed documents, store vectors, retrieve by cosine similarity, generate — is not the architecture that survives at scale. The gap between a RAG demo that impresses and a RAG system that a customer trusts is almost entirely a retrieval and grounding problem.

This post documents the patterns that survive. Chunking strategies, hybrid search, reranking, hallucination controls, eval harnesses, query rewriting — not as a survey of the literature but as a practitioner's account of what the failure modes look like in real systems and what the working fixes are. The code samples are real. The case studies are composites from production deployments. The cost figures are accurate as of June 2026.

What's actually happening

The RAG architecture in 2026

Retrieval-Augmented Generation has a deceptively simple architecture. The pipeline has three stages, and engineers who understand each stage precisely are better positioned to debug failures.

Ingest is the offline process that transforms raw documents into a searchable vector store. Documents are split into chunks. Each chunk is passed through an embedding model that produces a dense vector representation — a list of floating-point numbers that encodes semantic meaning in a high-dimensional space. Those vectors, along with their source metadata, are stored in a vector database. The quality of every downstream retrieval decision is bounded by the quality of the ingest step. If chunks are badly formed, no amount of retrieval sophistication will recover the information.

The ingest step is where most teams underinvest. Because it is an offline, one-time process, it feels cheap and reversible. In practice, the chunking decisions made during ingest bake assumptions into the shape of the data that are expensive to undo at query time. Changing your chunking strategy later means re-chunking every document, re-embedding every chunk, and rebuilding every index — a migration that can take hours or days on a large corpus and requires careful coordination to avoid a period where old and new embeddings coexist in the same vector store.

Retrieve is the online process that turns a user query into a ranked list of relevant chunks. The query is passed through the same embedding model used during ingest. The resulting query vector is compared against every stored chunk vector using a distance metric — almost universally cosine similarity or dot product — and the top-k closest chunks are returned. The retrieved chunks are the only context the language model has access to. Everything the LLM can say is constrained by what retrieval gives it.

A useful mental model for retrieval: it is not search in the traditional sense. Traditional keyword search asks "do these documents contain these words?" Embedding-based retrieval asks "are these documents close to this query in a high-dimensional semantic space?" Those are different questions with different failure modes. Keyword search fails when the user uses different vocabulary than the document. Embedding-based retrieval fails when two documents are semantically similar but one is relevant and one is not. At small corpus sizes, the nearest neighbor in embedding space is usually the right document because there is not much else nearby. At large corpus sizes, there are many semantically similar documents, and the nearest neighbor may not be the most factually relevant one.

Generate is the final step. A prompt is constructed from three components: a system instruction (the grounding prompt), the retrieved chunks (the context), and the user's original query. This prompt is submitted to the language model, which produces a response. The quality of the response is a product of retrieval quality times prompt quality times the model's capabilities — a chain where any weak link degrades the final output.

The generate step is where teams most often try to fix retrieval problems with prompt engineering. This does not work. A language model cannot answer a question from context that does not contain the answer, no matter how well-crafted the prompt. The grounding prompt can reduce hallucination; it cannot compensate for wrong chunks. When users report that the assistant is giving wrong answers, the first diagnostic step is always to inspect what the retrieval step is actually returning — not to iterate on the system prompt.

The four production failure modes

Production RAG systems fail in predictable ways. Understanding the failure mode is the prerequisite to fixing it.

Failure mode 1: Bad chunks. The most common root cause, and the easiest to underestimate. Arbitrary fixed-size tokenization — 512 tokens, split wherever the token count hits the limit — routinely breaks sentences in the middle, separates a table header from its rows, and cuts a numbered list at step 4 of 7. A chunk that ends mid-sentence has corrupted meaning. The embedding model will produce a vector that represents the fragment, not the concept. Retrieval will sometimes surface that chunk; the language model will see garbled input and either misinterpret it or hallucinate the missing context.

Failure mode 2: Bad retrieval. Cosine similarity is a measure of directional alignment in embedding space. It is not a measure of answer relevance. A document about "incident response procedures at financial institutions" is semantically close to a document about "incident response procedures at healthcare organizations" — both will have high cosine similarity to the query "what is our escalation procedure?" Neither may contain the organization-specific answer the user needs. This thematic-not-specific problem is endemic at scale: with 200 documents, your embedding space is sparse enough that the nearest neighbor is usually relevant. With 12,000 documents, the space is dense enough that the nearest neighbor may be thematically plausible but factually wrong.

Failure mode 3: Hallucination. Language models generate the most statistically likely continuation of a prompt. If the retrieved context is ambiguous, incomplete, or contradictory, the model will fill the gaps with information from its parametric memory — knowledge baked in during pretraining. That parametric knowledge may be wrong, outdated, or simply not applicable to the specific organization's policies. The critical point is that a grounding prompt ("answer only from the provided context") reduces but does not eliminate hallucination. Models under pressure to answer a question will find ways to do so, especially when the context is close-but-not-quite.

Failure mode 4: Citation drift. Citation drift is a subtler failure than hallucination, and it is particularly damaging in professional contexts where citations are the trust mechanism. The language model is given five chunks and asked to produce an answer with citations. The answer synthesizes information from chunks 1, 3, and 5, but the most plausible and fluent path to the answer runs through chunk 3 as the primary source. The model cites chunks 1 and 4 because they were present in the context and appeared relevant. The citation is structurally valid — those chunks were retrieved and presented — but factually misleading. The reader who traces the citation back to chunk 4 finds something tangentially related, not the source of the specific claim.

Diagnosing which failure mode you have

Before applying any fix, you need to know which failure mode is active. Applying the wrong remedy — improving chunking when the problem is citation drift, or hardening the grounding prompt when the problem is bad retrieval — wastes time and may mask the real issue.

The diagnostic procedure is mechanical:

1. Take a query that produced a wrong answer in production.
2. Inspect the retrieved chunks directly (log them, do not rely on the UI). Are the right chunks present in the top-5? If not: the problem is retrieval (bad chunks or bad search).
3. If the right chunks are present: read the generated answer and check each claim against the chunks. Are there claims not supported by any retrieved chunk? If yes: the problem is hallucination.
4. If there is no hallucination: check the citations. Does each cited chunk actually contain the claim attributed to it? If no: the problem is citation drift.
5. If the chunks are correct, no hallucination, and citations are accurate, but the answer is still wrong: check if the question requires synthesizing across more than 5 chunks. The problem may be top-k too low or context window ordering.

This five-step debug loop closes within 10 minutes for any given query. Run it on every user-reported wrong answer before making any system change. After 20–30 queries, patterns emerge: teams typically find that 70–80% of their wrong answers share one root cause, which dictates where to focus the engineering effort.

The pattern space

RAG manifests in several distinct application archetypes, and each has a characteristic failure signature.

Documentation and API reference bots

The characteristic query is precise and lexical: "what are the parameters for the createSession function?" or "what is the rate limit on the webhook endpoint?" These queries contain exact identifiers — function names, endpoint paths, parameter names — that are critical to correctness. Semantic similarity handles natural language well but tends to bury exact-match signals in embedding space. Two documents that contain the same identifier in different contexts will have similar embeddings to a query containing that identifier. The correct document is the one where the identifier is defined or documented, not merely mentioned.

This archetype is where hybrid search — combining vector similarity with lexical full-text matching — has the largest leverage. Exact identifier matches should boost relevance significantly. A pure vector approach will return the right document roughly 60–70% of the time for identifier-based queries; hybrid search consistently pushes that to 85–95% in production.

Customer support bots

The characteristic query is natural language, emotionally colored, and often underspecified: "my account is locked and I need to fix it now." The user is not thinking about the right technical vocabulary. The relevant chunks may be in an article titled "Account Recovery Procedures" which the user would never have searched for by name. This is where semantic similarity has its largest advantage over keyword search — the embedding space bridges the vocabulary gap between user language and documentation language.

The failure mode here is confidence without accuracy. A customer support bot that generates authoritative-sounding wrong answers is worse than no bot at all. The grounding and verification patterns are more important here than in documentation search.

Internal knowledge Q&A

Organizations with large document corpuses — policies, procedures, past decisions, research — use RAG to make that knowledge searchable. The characteristic challenge is temporal: documents have versions. A policy last updated in 2019 may still be in the corpus alongside its 2023 replacement. Without explicit temporal filtering or version metadata in the retrieval step, the language model may synthesize an answer from both versions, producing something that was never true.

Metadata filtering — filtering the vector search to chunks with last_modified > 2022-01-01, or filtering by a document_type attribute — is underused in this archetype. Vector similarity retrieval should almost always be combined with structured metadata filters for any corpus that has meaningful document-level attributes.

Code search and generation

Code repositories embed code as text. The embedding models trained on natural language do not represent code identifiers with the same fidelity as specialized code embedding models. The characteristic failure is retrieving code that is semantically similar (uses similar vocabulary, solves a similar problem class) but functionally different (uses a different API, operates on a different data structure, has different error handling). A code assistant that generates code from retrieved examples needs to retrieve the right examples, not just similar ones.

Code search benefits from the same hybrid approach as API docs — exact symbol matching via lexical search, semantic clustering via vector search — plus additional preprocessing: strip comments before embedding (they are often stale), embed function signatures separately from function bodies, and use language-aware chunking that respects function and class boundaries rather than token counts.

Putting the pipeline together: a TypeScript integration sketch

For teams running a Node.js backend, the RAG pipeline wires together cleanly with the OpenAI SDK, a pg client for pgvector, and a Cohere client for reranking. The following sketch shows the happy path — the pieces that follow are elaborated in detail in subsequent sections, but seeing the shape of the whole pipeline first is clarifying:

import OpenAI from "openai";
import { Pool } from "pg";
import Cohere from "cohere-ai";
 
const openai = new OpenAI({ apiKey: process.env.OPENAI_API_KEY });
const pool = new Pool({ connectionString: process.env.DATABASE_URL });
const cohere = new Cohere.CohereClient({ token: process.env.COHERE_API_KEY });
 
async function embedQuery(query: string): Promise<number[]> {
  const response = await openai.embeddings.create({
    model: "text-embedding-3-small",
    input: query,
  });
  return response.data[0].embedding;
}
 
async function hybridSearch(
  query: string,
  embedding: number[],
  topK: number = 50
): Promise<Array<{ id: string; content: string; metadata: Record<string, unknown>; score: number }>> {
  const embeddingStr = `[${embedding.join(",")}]`;
  // Run hybrid search (see SQL in later section)
  const result = await pool.query(
    `SELECT id::text, content, metadata, rrf_score AS score
     FROM hybrid_search($1::vector, $2, $3)`,
    [embeddingStr, query, topK]
  );
  return result.rows;
}
 
async function rerank(
  query: string,
  candidates: Array<{ id: string; content: string; metadata: Record<string, unknown>; score: number }>,
  topN: number = 5
): Promise<typeof candidates> {
  const response = await cohere.rerank({
    model: "rerank-v3.5",
    query,
    documents: candidates.map((c) => c.content),
    topN,
    returnDocuments: true,
  });
 
  return response.results.map((r) => ({
    ...candidates[r.index],
    score: r.relevanceScore,
  }));
}
 
async function generateAnswer(
  query: string,
  chunks: Array<{ id: string; content: string; metadata: Record<string, unknown>; score: number }>
): Promise<string> {
  const context = chunks
    .map((c) => `[CHUNK-${c.id}]\n${c.content}`)
    .join("\n---\n");
 
  const response = await openai.chat.completions.create({
    model: "gpt-4o",
    messages: [
      {
        role: "system",
        content: `Answer ONLY using the context below. Cite sources as [CHUNK-id].\nIf the context doesn't contain the answer, say so.\n\nContext:\n${context}`,
      },
      { role: "user", content: query },
    ],
    temperature: 0,
  });
  return response.choices[0].message.content ?? "";
}
 
// Full pipeline
export async function rag(query: string): Promise<string> {
  const embedding = await embedQuery(query);
  const candidates = await hybridSearch(query, embedding, 50);
  const reranked = await rerank(query, candidates, 5);
  return generateAnswer(query, reranked);
}

Each function in this sketch is a seam. The embedding model, retrieval strategy, reranker, and generation model are independently swappable. This is intentional — the single most common operational change in a production RAG system is swapping one of these components for a better one. Build your pipeline so those swaps are one-line changes, not architectural refactors.

The fix: a systematic approach

Chunking: from arbitrary splits to semantic units

The decision made at ingest time shapes everything downstream. A bad chunking strategy cannot be fixed at retrieval time.

Fixed-size chunking is where most projects start: split text every 512 tokens (or 256 or 1024), optionally with a 50-token overlap. It is trivial to implement and terrible in practice. The overlap helps with boundary effects — a concept split across a boundary will appear fully in at least one chunk — but it does not help when a table is split mid-row or a numbered procedure is split between steps 3 and 4. Use it for prototyping and replace it before production.

Sliding window chunking keeps fixed-size chunks but increases the overlap, typically to 20–30% of the chunk size. It generates more chunks for the same document, which increases storage and retrieval cost, but produces better boundary coverage. Still arbitrary in terms of semantic boundaries.

Semantic chunking is the right baseline for production. The approach: split on natural linguistic boundaries (sentences, paragraphs) and then merge adjacent units until an embedding distance threshold is crossed. When consecutive sentences have embeddings that are far apart in embedding space, that signals a topic transition — a natural chunk boundary. When they are close, they belong in the same chunk. The practical implementation uses sentence tokenization (spaCy, NLTK, or a simple regex that handles common abbreviations) followed by incremental embedding comparison.

import spacy
import numpy as np
from openai import OpenAI
 
nlp = spacy.load("en_core_web_sm")
client = OpenAI()
 
def embed(texts: list[str]) -> list[list[float]]:
    response = client.embeddings.create(
        model="text-embedding-3-small",
        input=texts
    )
    return [item.embedding for item in response.data]
 
def cosine_similarity(a: list[float], b: list[float]) -> float:
    a, b = np.array(a), np.array(b)
    return float(np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b)))
 
def semantic_chunk(
    text: str,
    max_tokens: int = 600,
    similarity_threshold: float = 0.75
) -> list[str]:
    doc = nlp(text)
    sentences = [sent.text.strip() for sent in doc.sents if sent.text.strip()]
 
    if not sentences:
        return []
 
    # Embed all sentences in batch
    embeddings = embed(sentences)
 
    chunks = []
    current_chunk_sentences = [sentences[0]]
    current_chunk_tokens = len(sentences[0].split())
 
    for i in range(1, len(sentences)):
        sim = cosine_similarity(embeddings[i - 1], embeddings[i])
        sentence_tokens = len(sentences[i].split())
 
        # Start a new chunk if: topic shift detected, or chunk would exceed max_tokens
        if sim < similarity_threshold or current_chunk_tokens + sentence_tokens > max_tokens:
            chunks.append(" ".join(current_chunk_sentences))
            current_chunk_sentences = [sentences[i]]
            current_chunk_tokens = sentence_tokens
        else:
            current_chunk_sentences.append(sentences[i])
            current_chunk_tokens += sentence_tokens
 
    if current_chunk_sentences:
        chunks.append(" ".join(current_chunk_sentences))
 
    return chunks

The similarity_threshold parameter is worth tuning for your corpus. A threshold of 0.75 produces coarser chunks (fewer, larger) and works well for narrative text. A threshold of 0.85 produces finer chunks and works better for reference documentation where concepts change frequently.

Layout-aware chunking handles structured documents — PDFs, HTML, Markdown, Word documents — where the visual or semantic structure matters. A PDF table that gets flattened to a string of space-separated values is nearly uninterpretable in a chunk. A section with a heading and body should usually be kept together. A code block should not be split mid-function.

The practical tools here are unstructured (the Python library from Unstructured.io) for PDFs with complex layouts, and MarkdownHeaderTextSplitter in LangChain for Markdown documents. For HTML, a parser that respects <table>, <section>, <pre>, and <code> tags before chunking is significantly better than raw text extraction.

from unstructured.partition.pdf import partition_pdf
from unstructured.chunking.title import chunk_by_title
 
# Layout-aware PDF processing
elements = partition_pdf(
    filename="document.pdf",
    strategy="hi_res",           # OCR + layout detection
    infer_table_structure=True,  # Tables extracted as structured data
    extract_images_in_pdf=False
)
 
# Chunk respecting title boundaries (headings become chunk anchors)
chunks = chunk_by_title(
    elements,
    max_characters=2000,
    new_after_n_chars=1500
)
 
for chunk in chunks:
    print(chunk.metadata.filename)
    print(chunk.text[:200])

Late chunking is an emerging approach (as of 2026) where the full document is embedded as a unit, and chunking happens at query time rather than ingest time. The intuition is that an embedding model with a long context window can attend to the full document when computing token-level representations, and then those representations can be pooled over query-relevant spans. The advantage is that chunk boundaries are chosen to match the query rather than being fixed at ingest. The disadvantage is that it requires a model that produces contextualized token embeddings (not just document-level embeddings) and significantly increases query-time compute. JinaAI's jina-embeddings-v3 and some Voyage AI models support this mode. It is worth watching but not production-default in most applications yet.

Embedding model choice

The embedding model translates text to vectors. The choice of model determines the geometry of your embedding space — which documents are near to each other, and therefore which are retrieved for which queries. This is not a decision to make once and forget: the MTEB (Massive Text Embedding Benchmark) leaderboard at huggingface.co/spaces/mteb/leaderboard is updated continuously and rankings shift as new models are released.

The operational landscape as of June 2026:

OpenAI text-embedding-3-small (1536 dimensions, $0.020 per million tokens) is the sensible default for most English-language applications. It is fast, cheap, accurate, and has broad language model compatibility since the same provider powers much of the generation stack. It is a reasonable starting point and often good enough for launch.

OpenAI text-embedding-3-large (3072 dimensions, $0.130 per million tokens) offers marginal retrieval improvement on most benchmarks — typically 2–5 percentage points on MTEB retrieval tasks — at 6.5× the cost. The improvement is real and meaningful in high-stakes applications; it rarely justifies itself in cost-sensitive ones.

Cohere embed-v3 (1024 dimensions with input-type specialization) is notable for explicit query/document asymmetry: you embed queries with input_type="search_query" and documents with input_type="search_document". This distinction encodes the asymmetry of the retrieval task directly into the model and produces measurably better results on asymmetric retrieval benchmarks. Cohere's multilingual variant handles 100+ languages.

Voyage AI (various dimensions; voyage-3-large, voyage-3) consistently ranks at the top of the MTEB retrieval leaderboard as of mid-2026. Voyage AI benchmarks show Voyage-3-large outperforming OpenAI text-embedding-3-large on several standard retrieval benchmarks. For applications where retrieval quality is the primary constraint and the embedding cost is not prohibitive, Voyage is worth a head-to-head evaluation against your specific corpus.

Open-source options: bge-large-en-v1.5 (BAAI) and gte-large (Alibaba DAMO) are strong open-source alternatives that can be self-hosted. They lag slightly behind the commercial leaders on MTEB retrieval tasks but are competitive in the 80th percentile. For applications with data sovereignty requirements or large embedding volumes where API cost is significant, self-hosting a bge-large-en-v1.5 on an H100-backed inference endpoint is economically compelling at scale.

A critical operational note: your embedding model version is a schema dependency. The vectors stored in your vector database are only comparable to vectors produced by the same model version. When OpenAI releases a new version of an embedding model, or when you decide to switch providers, you must re-embed the entire corpus. This is not a trivial operation on a large corpus. Plan for it by treating the embedding model as a versioned, schema-bound component with migration tooling.

The vector database

The vector database stores your embeddings and serves nearest-neighbor queries. The choice should be driven by your existing infrastructure, scale requirements, and operational overhead tolerance.

pgvector (the Postgres extension) is the correct default for most teams already running Postgres. It is free, integrated into your existing operational toolkit, and handles up to roughly one million vectors with acceptable performance given proper indexing. The SQL interface means you can combine vector search with arbitrary structured filtering in a single query — a major practical advantage.

-- Enable pgvector
CREATE EXTENSION IF NOT EXISTS vector;
 
-- Table schema with metadata columns for filtering
CREATE TABLE chunks (
    id          UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    document_id UUID NOT NULL REFERENCES documents(id) ON DELETE CASCADE,
    content     TEXT NOT NULL,
    embedding   vector(1536),  -- Match your embedding model's dimensionality
    chunk_index INTEGER NOT NULL,
    metadata    JSONB DEFAULT '{}',
    created_at  TIMESTAMPTZ DEFAULT NOW()
);
 
-- HNSW index for approximate nearest neighbor search
-- HNSW is preferred over IVFFlat for most production use cases:
-- better recall, faster query, slower build
CREATE INDEX ON chunks USING hnsw (embedding vector_cosine_ops)
    WITH (m = 16, ef_construction = 64);
 
-- Also a GIN index on metadata for fast JSON filtering
CREATE INDEX ON chunks USING gin (metadata);
 
-- Basic vector similarity query
SELECT
    c.id,
    c.content,
    c.metadata,
    1 - (c.embedding <=> $1::vector) AS similarity
FROM chunks c
WHERE c.metadata->>'document_type' = 'policy'   -- structured filter
  AND c.created_at > '2022-01-01'               -- temporal filter
ORDER BY c.embedding <=> $1::vector              -- cosine distance (ascending)
LIMIT 10;

The <=> operator is cosine distance (lower = more similar). The <-> operator is Euclidean distance. For normalized vectors — which all modern embedding models produce — cosine and Euclidean distance produce equivalent orderings, so the choice is cosmetic.

Pinecone is the managed vector database with the longest production track record. It handles tens of millions of vectors with minimal operational overhead, supports metadata filtering, and has reliable latency SLAs. At scale, the cost ($70+/month for the standard plan as of June 2026, scaling with index size and query volume) is often justifiable for production workloads. For prototyping, it is overkill.

Qdrant and Weaviate are open-source, self-hosted options with cloud-managed variants. Both have strong performance characteristics, support hybrid search natively, and are actively maintained. Qdrant's filtering implementation is particularly efficient and often outperforms pgvector for large-scale filtered queries. The operational overhead of self-hosting is real — plan for monitoring, backups, and upgrade procedures.

LanceDB is a file-based vector database appropriate for local development and prototyping. It requires no server process, stores data as Lance files on disk, and integrates with Python natively. Do not use it in production behind a multi-process web server.

Hybrid search: the most impactful production fix

The single most consistent improvement in RAG production deployments — across case studies, across teams, across application archetypes — is adding lexical full-text search alongside vector search and combining their results. This is hybrid search.

The intuition is direct: semantic similarity captures meaning but loses precision on specific identifiers. Full-text search captures exact matches but misses synonyms and paraphrase. A query for "createSession function parameters" should retrieve documents that contain "createSession" with high lexical precision (BM25) AND documents that discuss session initialization concepts (vector similarity). Neither alone is as good as both together.

In pgvector, hybrid search is implemented by running vector and full-text searches separately and combining their results. The standard combination approach is Reciprocal Rank Fusion (RRF), which is robust to different score scales and does not require tuning a weighting parameter.

-- Full-text search index (run once at setup)
ALTER TABLE chunks ADD COLUMN content_tsvector tsvector
    GENERATED ALWAYS AS (to_tsvector('english', content)) STORED;
CREATE INDEX ON chunks USING gin (content_tsvector);
 
-- Hybrid search with RRF combination
-- $1 = query vector, $2 = query text, $3 = RRF k constant (60 is standard)
WITH vector_results AS (
    SELECT
        id,
        ROW_NUMBER() OVER (ORDER BY embedding <=> $1::vector) AS rank
    FROM chunks
    ORDER BY embedding <=> $1::vector
    LIMIT 50
),
text_results AS (
    SELECT
        id,
        ROW_NUMBER() OVER (ORDER BY ts_rank(content_tsvector, query) DESC) AS rank
    FROM chunks, to_tsquery('english', $2) AS query
    WHERE content_tsvector @@ query
    LIMIT 50
),
rrf AS (
    SELECT
        COALESCE(v.id, t.id) AS id,
        COALESCE(1.0 / ($3 + v.rank), 0) +
        COALESCE(1.0 / ($3 + t.rank), 0) AS rrf_score
    FROM vector_results v
    FULL OUTER JOIN text_results t ON v.id = t.id
)
SELECT
    c.id,
    c.content,
    c.metadata,
    r.rrf_score
FROM rrf r
JOIN chunks c ON c.id = r.id
ORDER BY r.rrf_score DESC
LIMIT 10;

The RRF constant $3 (conventionally 60) controls how much rank position matters relative to presence in a result set. Higher values smooth out rank differences; lower values amplify them. 60 is a well-validated default from the information retrieval literature.

For full-text queries, the to_tsquery function in Postgres requires a specific syntax (ampersand for AND, pipe for OR, not free text). In practice, you want to transform the user's natural language query into a tsquery-compatible format. A simple approach: split on whitespace, filter stopwords, join with &. A more robust approach: use the language model to extract key terms from the query before constructing the full-text search.

import re
import psycopg2
from openai import OpenAI
 
STOPWORDS = {"a", "an", "the", "is", "are", "was", "were", "be", "been",
             "have", "has", "had", "do", "does", "did", "will", "would",
             "could", "should", "may", "might", "shall", "can", "for",
             "and", "or", "but", "in", "on", "at", "to", "of", "with"}
 
def query_to_tsquery(query: str) -> str:
    """Convert natural language query to Postgres tsquery format."""
    tokens = re.findall(r'\b[a-zA-Z0-9_]+\b', query.lower())
    terms = [t for t in tokens if t not in STOPWORDS and len(t) > 2]
    if not terms:
        return None
    return " & ".join(terms)
 
def hybrid_search(
    query: str,
    embedding: list[float],
    conn,
    top_k: int = 10,
    rrf_k: int = 60
) -> list[dict]:
    tsquery = query_to_tsquery(query)
 
    with conn.cursor() as cur:
        if tsquery:
            cur.execute("""
                WITH vector_results AS (
                    SELECT id, ROW_NUMBER() OVER (ORDER BY embedding <=> %s::vector) AS rank
                    FROM chunks ORDER BY embedding <=> %s::vector LIMIT 50
                ),
                text_results AS (
                    SELECT id, ROW_NUMBER() OVER (
                        ORDER BY ts_rank(content_tsvector, to_tsquery('english', %s)) DESC
                    ) AS rank
                    FROM chunks
                    WHERE content_tsvector @@ to_tsquery('english', %s)
                    LIMIT 50
                ),
                rrf AS (
                    SELECT
                        COALESCE(v.id, t.id) AS id,
                        COALESCE(1.0 / (%s + v.rank), 0) +
                        COALESCE(1.0 / (%s + t.rank), 0) AS rrf_score
                    FROM vector_results v FULL OUTER JOIN text_results t ON v.id = t.id
                )
                SELECT c.id, c.content, c.metadata, r.rrf_score
                FROM rrf r JOIN chunks c ON c.id = r.id
                ORDER BY r.rrf_score DESC LIMIT %s
            """, (embedding, embedding, tsquery, tsquery, rrf_k, rrf_k, top_k))
        else:
            # Fall back to pure vector search if no useful terms
            cur.execute("""
                SELECT id, content, metadata,
                       1 - (embedding <=> %s::vector) AS rrf_score
                FROM chunks
                ORDER BY embedding <=> %s::vector
                LIMIT %s
            """, (embedding, embedding, top_k))
 
        rows = cur.fetchall()
        return [
            {"id": str(r[0]), "content": r[1], "metadata": r[2], "score": float(r[3])}
            for r in rows
        ]

Reranking: the second stage that earns its latency

Hybrid search retrieves a candidate set — typically 50 candidates. Reranking uses a more expensive but more accurate model to re-score those candidates and select the final top-k for the prompt. The improvement is consistent and substantial: cross-encoder rerankers outperform bi-encoder (embedding-based) retrieval on most retrieval benchmarks because they attend jointly to the query and each candidate document, rather than computing independent embeddings.

The operational pattern is: retrieve 50 candidates cheaply (vector + BM25), rerank to 5 expensively (cross-encoder or API call). The latency cost of reranking is real — typically 100–400ms additional for an API-based reranker — but the accuracy improvement justifies it in most production contexts.

Cohere Rerank-3 and Voyage Rerank-2 are the leading managed reranking APIs as of June 2026. Cohere Rerank-3 is priced at approximately $1 per 1,000 searches (at standard top-k settings). For a RAG application serving 10,000 queries per day, that is $300 per month — a real cost but typically less than the cost of the language model calls.

import cohere
 
co = cohere.Client(api_key="YOUR_COHERE_API_KEY")
 
def rerank(
    query: str,
    candidates: list[dict],
    top_n: int = 5
) -> list[dict]:
    """
    Rerank candidates using Cohere Rerank-3.
    Candidates is a list of dicts with 'content' and 'id' keys.
    Returns the top_n candidates in reranked order.
    """
    if not candidates:
        return []
 
    documents = [c["content"] for c in candidates]
 
    response = co.rerank(
        model="rerank-v3.5",
        query=query,
        documents=documents,
        top_n=top_n,
        return_documents=True
    )
 
    reranked = []
    for result in response.results:
        original = candidates[result.index]
        reranked.append({
            **original,
            "rerank_score": result.relevance_score
        })
 
    return reranked

For teams who want to avoid the API dependency and latency of a managed reranker, cross-encoder models can be self-hosted. cross-encoder/ms-marco-MiniLM-L-6-v2 from Hugging Face is a strong baseline that runs on CPU at acceptable throughput for moderate query volumes:

from sentence_transformers import CrossEncoder
 
model = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")
 
def rerank_local(
    query: str,
    candidates: list[dict],
    top_n: int = 5
) -> list[dict]:
    pairs = [(query, c["content"]) for c in candidates]
    scores = model.predict(pairs)
 
    ranked = sorted(
        zip(candidates, scores),
        key=lambda x: x[1],
        reverse=True
    )
 
    return [
        {**candidate, "rerank_score": float(score)}
        for candidate, score in ranked[:top_n]
    ]

Hallucination control: grounding prompts and verification

The language model is the last mile. After retrieval and reranking deliver the right chunks, the prompt must constrain the model to use them faithfully.

The grounding prompt is the single most important hallucination control. It must be explicit, near the top of the system prompt, and not undercut by other instructions:

SYSTEM_PROMPT = """You are a helpful assistant that answers questions based on provided context documents.
 
CRITICAL RULES:
1. Answer ONLY using information from the provided context below.
2. If the context does not contain enough information to answer the question confidently, respond: "I don't have enough information in the provided documents to answer this question."
3. Do not use general knowledge or information from outside the provided context.
4. Cite the source of every claim using the chunk ID format [CHUNK-id].
5. If different sources contradict each other, note the contradiction rather than resolving it.
 
Context documents:
{context}
"""

The "I don't have enough information" escape hatch is critical. A model that is allowed to say "I don't know" will say it. A model that is pressured to always provide an answer will hallucinate. In customer-facing applications where an incomplete answer is better than a wrong answer, the escape hatch should be prominently permitted and not penalized in evaluation.

Citation enforcement asks the model to produce structured citations alongside its answer. The practical format is inline chunk IDs: every claim should be followed by [CHUNK-{id}]. This does not eliminate citation drift — the model can still cite the wrong chunk for a claim — but it creates an auditable artifact that enables verification.

def build_rag_prompt(
    query: str,
    chunks: list[dict],
    system_prompt_template: str = SYSTEM_PROMPT
) -> list[dict]:
    context_blocks = []
    for i, chunk in enumerate(chunks):
        context_blocks.append(
            f"[CHUNK-{chunk['id']}]\n"
            f"Source: {chunk['metadata'].get('source', 'Unknown')}\n"
            f"Content: {chunk['content']}\n"
        )
 
    context = "\n---\n".join(context_blocks)
    system = system_prompt_template.format(context=context)
 
    return [
        {"role": "system", "content": system},
        {"role": "user", "content": query}
    ]

The verification pass is a second language model call that checks each cited claim against its cited chunk. It is expensive — it doubles the LLM call count — but for applications where citation accuracy is a trust-critical requirement, it is worth it.

VERIFICATION_PROMPT = """You are a fact-checking assistant.
 
Given a CLAIM and a SOURCE DOCUMENT, determine whether the source document supports the claim.
 
CLAIM: {claim}
 
SOURCE DOCUMENT:
{source}
 
Respond with EXACTLY one of:
- SUPPORTED: [brief explanation]
- NOT_SUPPORTED: [brief explanation]
- PARTIALLY_SUPPORTED: [brief explanation]
 
Do not add commentary beyond the verdict and explanation."""
 
def verify_citation(
    claim: str,
    chunk_content: str,
    client: OpenAI
) -> dict:
    response = client.chat.completions.create(
        model="claude-sonnet-4-5",  # Fast, cheap, accurate for this task
        messages=[{
            "role": "user",
            "content": VERIFICATION_PROMPT.format(
                claim=claim,
                source=chunk_content
            )
        }],
        max_tokens=200,
        temperature=0
    )
 
    verdict_text = response.choices[0].message.content.strip()
    verdict = "UNKNOWN"
    for v in ["SUPPORTED", "NOT_SUPPORTED", "PARTIALLY_SUPPORTED"]:
        if verdict_text.startswith(v):
            verdict = v
            break
 
    return {"verdict": verdict, "explanation": verdict_text}

The verification pass catches citation drift: when the model cites chunk A for a claim but the claim is actually supported (or not supported at all) by the chunks, the verification will flag it. In production, a NOT_SUPPORTED verdict triggers either a reformulation of the answer (remove the unsupported claim) or a user-visible caveat ("this claim could not be verified in the source documents").

Query rewriting: bridging the query-document vocabulary gap

User queries are short, ambiguous, and conversational. Chunks are dense, structured, and authoritative. The vocabulary gap between them is a fundamental retrieval problem. Two approaches close it.

HyDE (Hypothetical Document Embeddings) inverts the embedding direction. Instead of embedding the user's query and finding similar documents, you ask the language model to generate a hypothetical document that would answer the query, then embed that hypothetical document and find real documents similar to it. The intuition is that a generated document uses the same vocabulary and structure as the source corpus, closing the vocabulary gap.

HYDE_PROMPT = """Write a brief, factual passage (3–5 sentences) that directly answers the following question.
Write it as if it were an excerpt from a technical documentation page.
Do not hedge or qualify — write as if you have the answer.
 
Question: {query}
 
Passage:"""
 
def hyde_expand(query: str, client: OpenAI) -> str:
    """Generate a hypothetical document that would answer the query."""
    response = client.chat.completions.create(
        model="gpt-4o-mini",  # Cheap model for expansion; quality matters less than speed
        messages=[{
            "role": "user",
            "content": HYDE_PROMPT.format(query=query)
        }],
        max_tokens=300,
        temperature=0.3
    )
    return response.choices[0].message.content.strip()
 
def retrieve_with_hyde(
    query: str,
    embed_fn,
    search_fn,
    client: OpenAI,
    top_k: int = 10
) -> list[dict]:
    # Generate hypothetical document
    hypothetical = hyde_expand(query, client)
 
    # Embed the hypothetical document, not the original query
    hyp_embedding = embed_fn(hypothetical)
 
    # Retrieve using the hypothetical embedding
    return search_fn(hypothetical, hyp_embedding, top_k=top_k)

HyDE works well for queries where the user's phrasing is very different from how the answer would be stated in the source documents. It works less well when the query contains critical lexical signals (identifiers, names) that should be preserved exactly. In those cases, HyDE's generative step may rephrase the identifier, losing the exact-match signal. Hybrid HyDE — use HyDE embedding for the vector search component, use the original query terms for the full-text search component — is a practical compromise.

Multi-query retrieval runs the same query in three or more phrasings, retrieves candidates for each, and takes the union before reranking. The language model generates the query variants:

MULTI_QUERY_PROMPT = """Generate 3 different phrasings of the following question.
Each phrasing should capture the same information need but use different vocabulary.
Return exactly 3 phrasings, one per line, with no numbering or bullets.
 
Original question: {query}
 
Phrasings:"""
 
def multi_query_retrieve(
    query: str,
    embed_fn,
    search_fn,
    rerank_fn,
    client: OpenAI,
    candidates_per_query: int = 20,
    final_top_k: int = 5
) -> list[dict]:
    # Generate query variants
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": MULTI_QUERY_PROMPT.format(query=query)}],
        max_tokens=200,
        temperature=0.5
    )
    variants = [query] + response.choices[0].message.content.strip().split("\n")[:3]
 
    # Retrieve candidates for each variant
    seen_ids = set()
    all_candidates = []
 
    for variant in variants:
        embedding = embed_fn(variant)
        results = search_fn(variant, embedding, top_k=candidates_per_query)
        for r in results:
            if r["id"] not in seen_ids:
                seen_ids.add(r["id"])
                all_candidates.append(r)
 
    # Rerank the union over the original query
    return rerank_fn(query, all_candidates, top_n=final_top_k)

Multi-query retrieval is particularly effective for queries about topics with rich synonymy — where the user might say "error" and the documentation says "exception," or the user says "how do I" and the documentation uses "procedure for." The union across multiple phrasings dramatically improves recall.

Context window management

Retrieved chunks land in the language model's context window. The number and size of chunks is a direct tradeoff between cost, latency, and answer quality.

A practical baseline: 5 chunks × 500 tokens/chunk = 2,500 tokens of context. Combined with a typical system prompt (~500 tokens) and the user query (100 tokens), the total input to the model is around 3,100 tokens. At Anthropic Claude Sonnet-class pricing ($3 per million input tokens as of June 2026), that is less than $0.01 per query. Sustainable at high volume.

The temptation is to increase top-k to improve recall. Retrieving 20 chunks at 500 tokens each adds 10,000 tokens of context — a 3× cost increase for uncertain accuracy improvement. Research by Liu et al. (2023, "Lost in the Middle") demonstrates that language models attend disproportionately to information at the beginning and end of a long context. Information in the middle of a long context window is substantially less likely to be used in the generated answer. This is the "lost in the middle" problem.

The practical implications:

• Keep top-k at 5–8 for most applications. More is not better.
• If you must include more chunks, use the reranker's confidence scores to order them: highest-scoring chunk first, second-highest last. Put the most relevant chunks at the positions the model attends to most.
• For very long documents where a single document may need multiple chunks, deduplicate: if the top-3 chunks by reranking are all from the same source document, consider consolidating them into one block.

Cost modeling: where the money actually goes

Before adding every optimization in this post, it is worth understanding the cost breakdown of a production RAG pipeline. Most teams over-spend on the wrong things and under-invest in the right ones.

Embedding costs are typically negligible. OpenAI text-embedding-3-small costs $0.020 per million tokens as of June 2026. A corpus of 12,000 documents averaging 5,000 words each is approximately 60 million tokens total — a one-time ingest cost of $1.20. Re-embedding after a chunking strategy change: $1.20. Embedding 10,000 user queries per day at an average of 50 tokens each: $0.01 per day. Embedding is not the cost you need to optimize.

Vector database costs depend on your choice. pgvector is free (you are already paying for Postgres). Pinecone's standard plan starts at $70/month as of June 2026, plus query and write costs at scale. Qdrant Cloud has a similar pricing model. For most applications under 5 million vectors, pgvector with an HNSW index running on your existing database is the right choice and costs nothing incrementally.

Reranking costs are real but predictable. Cohere Rerank-3 is approximately $1 per 1,000 rerank requests as of June 2026. At 10,000 queries per day, that is $10/day — $300/month. For 100,000 queries per day, it is $3,000/month, which becomes a meaningful line item. At high query volumes, self-hosting a cross-encoder model (cross-encoder/ms-marco-MiniLM-L-6-v2 or similar) on a small GPU instance is economically compelling.

LLM generation costs are the dominant variable. The input token cost scales with context window size (number and size of chunks in the prompt). The output token cost scales with answer length. Using Anthropic Claude Sonnet-class models at ~$3 per million input tokens and ~$15 per million output tokens as of June 2026:

• A query with 5 chunks × 500 tokens = 2,500 context tokens, 600-token answer: (3,100 × $0.000003) + (600 × $0.000015) = $0.009 + $0.009 = ~$0.018 per query
• At 10,000 queries per day: ~$180/day = ~$5,400/month

Adding a verification pass doubles the LLM call count but not necessarily the cost — verification calls are shorter (checking one claim against one chunk) and can use a cheaper model. Using GPT-4o-mini for verification ($0.15/M input, $0.60/M output as of June 2026) at 3 verification calls per query:

• 3 × (1,000 tokens input × $0.00000015 + 200 tokens output × $0.0000006) = 3 × ($0.00015 + $0.00012) = ~$0.0008 per query
• Negligible relative to the generation cost.

The cost optimization levers, in order of impact: choose the right model tier for the task (use a smaller model for query rewriting and verification, reserve the large model for final generation); reduce context window size with better reranking (fewer, more relevant chunks costs less than more, less relevant chunks); cache responses for frequently repeated queries (a significant fraction of support bot queries are identical or near-identical).

The eval harness: the thing most teams skip

An eval harness is a held-out dataset of query/answer pairs used to measure retrieval and generation quality systematically. Without it, you are flying blind: every change to chunking strategy, embedding model, retrieval parameters, or prompt is an untested hypothesis.

The minimal eval harness:

# eval/rag-eval-config.yaml
version: "1.0"
dataset: "eval/query-answer-pairs.jsonl"
metrics:
  - recall_at_k:
      k: [1, 3, 5, 10]
  - mrr:
      max_k: 10
  - answer_correctness:
      judge_model: "gpt-4o"
      prompt_template: "eval/judge-prompt.txt"
 
retrieval:
  embedding_model: "text-embedding-3-small"
  vector_db:
    type: "pgvector"
    connection_string: "${DATABASE_URL}"
  hybrid_search: true
  reranking:
    enabled: true
    model: "rerank-v3.5"
  top_k: 5
 
schedule: "nightly"
alert_on_regression:
  recall_at_5: 0.05   # Alert if recall@5 drops more than 5 percentage points
  answer_correctness: 0.05

The dataset format is straightforward:

# eval/generate-eval-dataset.py
# A script to generate the eval dataset from human-curated pairs.
# Each entry: {query, expected_answer, relevant_chunk_ids}
 
import json
from pathlib import Path
 
eval_pairs = [
    {
        "query": "What is the escalation procedure for a Tier-2 incident?",
        "expected_answer": "For a Tier-2 incident, the engineer on-call should...",
        "relevant_chunk_ids": ["chunk-uuid-1", "chunk-uuid-2"],
        "category": "procedures"
    },
    # ... 50-200 more pairs
]
 
Path("eval/query-answer-pairs.jsonl").write_text(
    "\n".join(json.dumps(pair) for pair in eval_pairs)
)

The LLM judge prompt evaluates answer correctness:

You are evaluating the quality of an AI assistant's answer.
 
QUESTION: {query}
EXPECTED ANSWER: {expected_answer}
ACTUAL ANSWER: {actual_answer}
 
Rate the actual answer on the following dimensions (1–5 scale):
1. FACTUAL_ACCURACY: Is the information correct?
2. COMPLETENESS: Does it address all key points in the expected answer?
3. HALLUCINATION: Does it include claims not supported by the expected answer? (5 = no hallucination, 1 = major hallucination)
4. CITATION_ACCURACY: Do the citations point to relevant sources?
 
Respond in JSON:
{"factual_accuracy": N, "completeness": N, "hallucination": N, "citation_accuracy": N, "explanation": "..."}

Run this harness nightly. Any regression in recall@5 or answer correctness beyond the alert threshold triggers a review before the change ships. The cadence is critical: without scheduled runs, the harness gets built and then gradually ignored as the team focuses on features.

Anti-patterns

These are the patterns that look reasonable in a prototype and cause production incidents.

1. Treating cosine similarity score as a fixed relevance threshold.

A cosine similarity above 0.85 does not mean the chunk is relevant. It means the chunk is directionally close to the query in the embedding model's latent space. The calibration of that threshold is specific to the embedding model, the corpus, and the query distribution. A threshold calibrated on OpenAI text-embedding-3-small will not transfer to Voyage-3-large. A threshold calibrated on your 200-document demo corpus will not transfer to your 12,000-document production corpus. Use the eval harness to find the right threshold empirically, and plan to recalibrate it when any of those three factors changes.

2. Using the embedding model as a black box with no eval harness.

The embedding model is a dependency, not a utility. When you switch embedding models (which you will, as better models become available), all existing vectors in your database are now computed in a different space and are no longer comparable to new vectors. A mixed-model vector database produces undefined retrieval behavior — some queries will find old-model vectors, some will find new-model vectors, and the cosine distance between them is meaningless. The mitigation is: versioning on the chunks table, a reindex process, and an eval harness to verify the new model before deploying it.

3. No reranking.

Cosine-based retrieval alone rarely beats hybrid search plus reranking in production. The improvement from adding reranking is consistent enough across applications that it should be the default architecture, not an optimization to add later. The latency cost is real and manageable; the accuracy improvement often justifies the cost even at launch.

4. Ingesting raw PDFs without layout extraction.

A PDF table converted to plain text by a naive extractor produces something like: Column A Column B Column C Value 1 Value 2 Value 3 Value 4 Value 5 Value 6. The column associations are lost. An embedding model given this text produces a vector that represents a list of tokens, not a structured table. The chunk will be retrieved for queries about any of those values and will be useless as context because the structure is gone. Use unstructured, pdfplumber, or camelot for layout-aware PDF extraction before chunking.

5. Treating a chunk as an answer.

A chunk is a retrieval unit. It is chosen to be small enough for efficient embedding and retrieval. It is not chosen to be a complete, self-contained answer. The language model's job is to synthesize an answer from multiple chunks — to do what a human expert would do when consulting multiple relevant passages of documentation. A RAG system that outputs the top chunk as the answer, bypassing the generation step, is not using the architecture correctly and will frequently produce incomplete or decontextualized responses.

6. Caching embeddings indefinitely.

Embeddings are vectors produced by a specific model version. If you re-embed a document with a new model version and your cache serves the old vector, queries will silently degrade because they are being compared against vectors in a different latent space. Embedding caches should be keyed on (text, model_id, model_version) and invalidated when the model version changes. This is not a theoretical concern — OpenAI has released multiple versions of text-embedding-3-small with different behavior.

7. Putting all retrieved chunks into the prompt in retrieval order without ranking.

If your retrieval step returns chunks sorted by cosine score and you insert them into the prompt in that order, you are placing the highest-relevance chunk at position 1 (good) and the progressively less relevant chunks after it. Due to the "lost in the middle" effect, the most relevant chunk gets full attention, but chunks 3, 4, 5 receive systematically less attention than chunks 1 and 5. The correct ordering for a long context: highest-relevance at position 1, second-highest at the last position, remaining chunks in between. This is counterintuitive but empirically supported.

Case studies

Case study 1: Customer support RAG over 12,000 internal documents

Context: A team built a RAG-powered support assistant for internal customer support agents. The corpus was 12,000 documents: product manuals, SOPs, policy documents, historical incident write-ups, and internal wikis. Documents ranged from 1 page to 80 pages.

Initial architecture: Fixed 512-token chunks with 50-token overlap. OpenAI text-embedding-3-small. pgvector with IVFFlat index. Cosine kNN, top-5. No reranking. Grounding prompt present but minimal.

Initial performance (eval harness, 150 hand-curated query/answer pairs):

• Recall@5: 47% (the correct chunk was in the top-5 results for 47% of queries)
• Hallucination rate (LLM judge, checking answer claims against retrieved chunks): 12%
• Answer correctness (LLM judge, holistic): 52%

Root cause analysis: Two failure modes dominated. First, the fixed-size chunking was splitting multi-step procedures mid-step and separating table headers from their rows. This produced chunks that were individually high-similarity to queries about those procedures but missing critical context. Second, the cosine-only retrieval was surfacing thematically similar but not specifically relevant documents — queries about "escalation procedure" were retrieving chunks from all documents that discussed escalation in any context, not the specific SOP that applied.

Changes applied (in order of impact):

Step 1: Semantic chunking + layout-aware PDF processing. Replaced fixed-size chunks with semantic chunking for narrative documents and layout-aware extraction for PDFs. Average chunk size increased from 512 tokens (hard ceiling) to ~380 tokens (average, with semantic boundaries). Total chunk count decreased by 22% because semantic chunks are more information-dense. Recall@5 after this change alone: 59%.

Step 2: Hybrid search (pgvector + tsvector + RRF). Added full-text search alongside vector search, combined with RRF at k=60. This change was particularly impactful for queries containing specific product names, policy identifiers, and acronyms (the support team used acronyms pervasively; the documentation used the same acronyms). Recall@5: 71%.

Step 3: Cohere Rerank-3, top-50 → top-5. Added reranking as the final retrieval stage. Recall@5: 78%. Hallucination rate dropped from 12% to 7% (a consequence of better retrieval — the model was seeing more relevant context and less temptation to fill gaps).

Step 4: Grounding prompt hardening + citation enforcement. Rewrote the system prompt with explicit "answer only from context" language and required inline citations. Added a verification pass for answers above a risk threshold (queries categorized as "policy" or "procedure" by a classifier). Hallucination rate: 3%.

Final performance:

• Recall@5: 78% (from 47%)
• Hallucination rate: 3% (from 12%)
• Answer correctness: 81% (from 52%)
• Latency P50: 1.8s (from 0.9s — the reranking and verification pass added latency)
• Cost per query: ~$0.008 (embeddings + reranking + generation)

The latency increase was acceptable for the internal support agent use case, where the alternative was a 5-minute search through 12,000 documents. The customer's support team reported materially faster query resolution in production.

Case study 2: API reference documentation bot

Context: A developer tools company built a documentation assistant over their API reference. The corpus was approximately 2,000 pages of API documentation including function signatures, parameter tables, code examples, and conceptual guides.

The characteristic failure: Queries containing specific function or method names — "createSession", "parseWebhookPayload", "retry_on_rate_limit" — were returning results that mentioned those identifiers in passing (in code examples, in "see also" sections) rather than the primary definition pages. Semantic similarity alone could not distinguish "the page that defines createSession" from "a page that uses createSession in an example."

Initial performance: Citation accuracy 81% (the cited chunk was actually the source of the claimed information). Recall@5 for identifier-based queries: 63%.

Fix: Hybrid search with a BM25 boost specifically on exact-match of code identifiers. The implementation extracted code identifiers from the query (tokens matching [a-zA-Z_][a-zA-Z0-9_]* with camelCase or snake_case patterns), then applied a higher weight to lexical matches on those identifiers in the RRF combination. Additionally, pages were chunked with function signatures kept in the same chunk as their parameter tables and description — no function definition was ever split across chunks.

Additional change: The documentation had a rich structure (function name → signature → parameters → return value → examples → notes). This structure was preserved as metadata on each chunk. Retrieval was filtered by chunk_type when the query pattern was clearly about a specific element (e.g., a query beginning with a function name that matched the corpus was routed to chunks with chunk_type = "definition" first).

Results: Citation accuracy 81% → 96%. Recall@5 for identifier-based queries: 63% → 91%. Developer satisfaction (internal survey, 47 respondents): significantly positive shift toward "answers with correct citations I can verify."

Case study 3: Legal document RAG

Context: A legal tech application indexed case law — approximately 40,000 court opinions, averaging 15,000 words each. Users were lawyers asking complex, multi-part questions with dense legal terminology. Typical query length: 30–50 tokens.

The challenge: Legal questions are long, specific, and use vocabulary that does not always match the phrasing in court opinions. A question about "does the reasonable reliance standard apply when the representor has superior knowledge" requires retrieval of cases that may articulate the same principle using different phrasing — "superior knowledge of the representor," "justifiable reliance," "asymmetric information between parties."

Initial recall@5 (200 lawyer-curated test queries): 43%. This is catastrophically low for a legal research tool — lawyers missing 57% of relevant precedents is not a product, it is a liability.

Architecture applied:

Multi-query retrieval with HyDE: each query was expanded to three variants by the language model, plus a hypothetical document embedding. The union of four retrieval sets (original + 3 variants, each top-20) produced ~60 unique candidates. These were reranked with Voyage Rerank-2 to top-10.

Semantic chunking for long documents: each 15,000-word court opinion was chunked semantically, producing ~25–40 chunks per opinion. Chunks preserved the legal section structure (facts, procedural history, legal standard, analysis, holding) by using section headings as hard chunk boundaries, then semantic subdivision within sections.

Temporal and jurisdiction metadata filters: queries were analyzed to extract temporal constraints ("cases after 2015"), jurisdictional constraints ("federal circuit"), and case type constraints ("securities fraud"). These were applied as pre-filters before vector search.

Results after 200 lawyer-tested queries:

• 18% returned "I don't have enough information" (the model correctly identified questions that required case law not in the corpus)
• 76% of answered queries rated "correct or largely correct" by the lawyers
• 6% of answered queries contained hallucinated citations — citations to cases that did not contain the proposition attributed to them

The 6% hallucination rate was unacceptable to the legal team. It led directly to the implementation of the verification pass described above: every cited case is verified by a second language model call that checks whether the cited passage from the case supports the specific proposition attributed to it. Post-verification, hallucinated citations: 1.2%.

The verification pass added approximately 800ms to query latency (4–6 verification calls in parallel, one per cited source). For legal research, where the alternative is hours of manual research, this was acceptable.

Limits and caveats

Embedding models change quickly

The MTEB leaderboard is not a stable ranking. Models that lead in June 2026 may be surpassed by the time you read this. The operational implication is that your retrieval architecture needs to treat the embedding model as a replaceable component with a defined migration path — not as a fixed infrastructure choice. A reindex pipeline that can rebuild your entire vector store from raw chunks in a single automated job is not optional. It is part of the production readiness checklist.

Source document quality is the upper bound

A RAG system can only answer questions that are answerable from the source documents. If the documents are poorly written, internally inconsistent, outdated, or incomplete, the RAG system will faithfully reflect those problems — and a language model that is skilled at synthesizing confident-sounding prose will make the problems less visible, not more. The correct place to invest before deploying a RAG system is in the quality and completeness of the source corpus. Document hygiene — removing outdated content, flagging internally contradictory documents, ensuring version metadata is accurate — yields more improvement than any retrieval optimization applied to a poor corpus.

Some questions cannot be answered by retrieval

RAG is a pattern for grounding language model generation in a specific corpus. It is not appropriate for questions that require synthesis across many documents simultaneously (summarize all 1,200 quarterly reports), questions that require real-time information not in the corpus, or questions that require complex multi-step reasoning that a single retrieval pass cannot support. For multi-hop reasoning — "find the policy that applies to contractors in the EU, then find the specific provision that covers intellectual property rights" — a single-pass RAG will often fail. Agentic retrieval patterns (the model decides when to retrieve, what to retrieve, and can issue multiple retrieval queries in sequence) are the right tool for those cases, at the cost of significantly more complex implementation and higher latency.

Hallucination is not fully solved

The patterns in this post reduce hallucination substantially. They do not eliminate it. Any RAG system deployed in a context where a wrong answer has meaningful consequences — medical information, legal advice, financial guidance, safety-critical operations — must include human oversight in the loop, must communicate uncertainty clearly to users, and must not be positioned as authoritative without human review. The verification pass is a mechanical check, not a guarantor of truth. A model can produce a verified citation that accurately represents what the source document says, and the source document can be wrong.

The "lost in the middle" mitigation has limits

Reordering chunks (highest relevance first and last) helps but does not fully recover attention to middle-position context. For applications requiring full attention to all retrieved context, a smaller top-k with higher individual chunk quality is more reliable than a large top-k with context reordering. The research by Liu et al. on this topic is based on models as of 2023; current models with explicit long-context training have somewhat improved middle-position attention, but the effect remains measurable.

Sources and further reading

The patterns in this post are grounded in primary sources. The links and citations below are the starting points for deeper investigation.

OpenAI Embeddings Guide — platform.openai.com/docs/guides/embeddings — The authoritative reference for OpenAI embedding models, including dimensionality, pricing, and best practices for normalization and distance metrics.

Anthropic Contextual Retrieval — Anthropic's blog post on contextual retrieval introduces the technique of prepending chunk-level context (generated by a language model) to each chunk before embedding, improving retrieval accuracy on their benchmarks. The post includes ablation results showing the impact of each component.

Cohere Rerank documentation — docs.cohere.com/docs/reranking — API documentation and integration guide for Cohere Rerank-3. Includes latency benchmarks and accuracy comparisons against cross-encoder alternatives.

pgvector documentation — github.com/pgvector/pgvector — The canonical reference for vector operations in Postgres: supported distance metrics, index types (IVFFlat vs HNSW), performance characteristics, and query syntax.

MTEB leaderboard — huggingface.co/spaces/mteb/leaderboard — The Massive Text Embedding Benchmark tracks embedding model performance across retrieval, classification, clustering, and other tasks. The retrieval subtask is the most directly relevant for RAG applications.

"Lost in the Middle: How Language Models Use Long Contexts" — Liu et al., 2023 (arXiv:2307.03172) — The paper that documented the attention boundary effect empirically. Required reading for anyone designing context window ordering strategies.

"Precise Zero-Shot Dense Retrieval without Relevance Labels" (HyDE) — Gao et al., 2022 (arXiv:2212.10496) — The original paper introducing Hypothetical Document Embeddings, with evaluation on standard retrieval benchmarks and ablation against query-only baselines.

LangChain RAG documentation — python.langchain.com/docs/use_cases/question_answering/ — LangChain's RAG tutorials cover the standard pipeline from document loading through generation, with code samples for most of the components described in this post.

LlamaIndex RAG documentation — docs.llamaindex.ai/en/stable/understanding/rag/ — LlamaIndex provides higher-level abstractions for RAG pipeline construction, including built-in hybrid search, reranking integrations, and eval frameworks.

Voyage AI benchmarks — blog.voyageai.com — Voyage AI publishes comparisons of their embedding models against OpenAI and Cohere benchmarks on standard retrieval tasks. The methodology sections are worth reading for understanding what MTEB retrieval benchmarks do and do not measure.

<!-- audit-checklist
- [ ] Every model performance claim has a primary source OR is scoped as observation.
- [ ] All pricing claims scoped to date of writing (2026-06).
- [ ] All model + version names verified.
- [ ] All linked URLs return HTTP 200.
- [ ] No proprietary client name used without consent.
- [ ] No vertical industry reveal.
- [ ] Code samples compile on cited library versions.
- [ ] Reading time matches body length to ±20 %.
-->