Inside our RAG pipeline: chunking, embeddings, and hybrid search

Retrieval-augmented generation gets pitched as a simple loop: embed your docs, embed the question, find the closest match, hand it to the model. In production, most of the accuracy comes from the parts that pipeline diagram leaves out.

Chunking is where most quality is won or lost

We don’t split content by a fixed token count. Headings, list boundaries, and table structure all inform where a chunk starts and ends, because a chunk that cuts a table in half is useless at retrieval time no matter how good the embedding model is. Each chunk also carries its source heading as metadata, so the model sees “Refunds → Annual Plans” even if the matched text itself doesn’t repeat that context.

Hybrid search, not just vector search

Pure vector similarity misses exact-match cases — a customer typing an error code or a plan name verbatim should hit that exact string, not just something semantically nearby. Every query runs through both a dense vector search and a sparse keyword search (BM25), and we merge the results with reciprocal rank fusion before reranking.

Reranking before generation

The top ~20 candidates from hybrid search go through a lightweight cross-encoder reranker before the final 4–6 chunks are selected for the prompt. This step alone measurably reduces “confidently wrong” answers, because it catches cases where a chunk is topically similar but doesn’t actually answer the question asked.

What this means for you

None of this requires configuration — it runs the same way for every workspace. The practical takeaway is upstream: well-structured source content (clear headings, one topic per section) will always retrieve better than a wall of undifferentiated text, no matter how good the pipeline underneath it is.