How to Build a Production-Ready RAG Pipeline from Scratch: A Step-by-Step Guide

How to Build a Production-Ready RAG Pipeline from Scratch: A Step-by-Step Guide

Retrieval-Augmented Generation (RAG) is an architecture that allows LLMs to answer questions based on external knowledge sources. By 2026, RAG has become the standard for chatbots, enterprise search engines, and decision support systems. But building a pipeline that works not only in a notebook but also under load is a non-trivial task.

In this guide, I'll break down a production-ready RAG pipeline: from chunking to reranking. No fluff, just code and benchmarks.

1. Concept: Why Simple RAG Doesn't Work in Production

A typical mistake beginners make is taking a standard pipeline: "split text -> send to embedding -> find top-5 -> feed to LLM." In production, this approach fails for three reasons:
- Poor chunking: long text chunks reduce search accuracy, short ones lose context.
- Homogeneous search: vector search (semantic) alone doesn't find exact matches (e.g., document numbers). BM25 alone doesn't understand synonyms.
- Lack of reranking: the first 5 results may be garbage, and the LLM will produce gibberish.

Production-ready RAG is a hybrid pipeline that combines semantic and lexical search, then reorders results with a reranker.

2. Architecture of a Production-Ready RAG Pipeline

The pipeline consists of 6 stages:

1. Ingestion (data loading): parsing documents (PDF, HTML, Markdown).
2. Chunking (splitting into chunks): intelligent splitting considering text structure.
3. Embedding (vectorization): converting chunks into embeddings.
4. Hybrid search: simultaneous search by vector and keywords.
5. Reranking: refining relevance order.
6. Generation: passing top-K chunks to the LLM.

3. Implementation: Code and Configuration

3.1. Chunking: Strategies and Benchmarks

Chunking is the most critical stage. Poor chunking kills search accuracy by 30-50%.

Three strategies:

Code example (Recursive chunking with LangChain):

from langchain.text_splitter import RecursiveCharacterTextSplitter

text_splitter = RecursiveCharacterTextSplitter(
    chunk_size=512,
    chunk_overlap=50,
    separators=["\n\n", "\n", ". ", " ", ""]
)

chunks = text_splitter.split_text(document)

Benchmark: On the QMSUM dataset (medical documentation cases), Recursive chunking with overlap=50 yields recall@5 = 0.87, while fixed-size yields 0.76.

3.2. Choosing an Embedding Model

In 2026, the top 3 models for production:
- text-embedding-3-large (OpenAI) — best quality, but expensive ($0.13/1M tokens).
- intfloat/multilingual-e5-large (Open Source) — excellent multilingual capability, free, but requires GPU.
- BAAI/bge-m3 (Open Source) — balance of speed and quality, supports dense and sparse embeddings.

Tip: For Russian, use intfloat/multilingual-e5-large — it achieves F1-score 0.91 on RuBQ (Russian QA dataset).

3.3. Hybrid Search: Dense + Sparse

Hybrid search combines results from vector (dense) and lexical (sparse) search. This improves recall@10 by 15-20%.

Implementation on Qdrant:

from qdrant_client import QdrantClient
from qdrant_client.http import models

client = QdrantClient(url="http://localhost:6333")

# Create collection with sparse vector support
client.create_collection(
    collection_name="docs",
    vectors_config={
        "dense": models.VectorParams(size=768, distance=models.Distance.COSINE),
        "sparse": models.VectorParams(size=0, distance=models.Distance.COSINE, sparse=True)
    }
)

# Search with fusion
results = client.query_points(
    collection_name="docs",
    query=models.FusionQuery(
        fusion=models.Fusion.RRF,  # Reciprocal Rank Fusion
        queries=[
            models.NearestQuery(nearest=[0.1, 0.2, ...]),  # dense
            models.NearestQuery(nearest=[0.0, 0.0, 0.3, ...])  # sparse
        ]
    ),
    limit=20
)

Alternative: Elasticsearch with dense + sparse (Elastic Learned Sparse Encoder).

3.4. Reranking: How to Improve Accuracy by 10-15%

A reranker is a separate model that reorders search results. Unlike embeddings, a reranker considers the mutual position of query and document.

Top models:
- BAAI/bge-reranker-v2-m3 — open source, F1-score 0.94 on MS MARCO.
- Cohere rerank — paid, but very accurate (0.97).

Example with BAAI/bge-reranker-v2-m3:

from transformers import AutoModelForSequenceClassification, AutoTokenizer

model = AutoModelForSequenceClassification.from_pretrained("BAAI/bge-reranker-v2-m3")
tokenizer = AutoTokenizer.from_pretrained("BAAI/bge-reranker-v2-m3")

def rerank(query, documents, top_k=5):
    pairs = [[query, doc] for doc in documents]
    inputs = tokenizer(pairs, padding=True, truncation=True, return_tensors="pt")
    scores = model(**inputs).logits.squeeze(-1).detach().numpy()
    ranked_indices = scores.argsort()[::-1][:top_k]
    return [documents[i] for i in ranked_indices]

Benchmark: On the FIQA dataset (financial questions), reranking increases NDCG@10 from 0.67 to 0.83.

4. Production: Scaling and Monitoring

4.1. Caching

Cache embeddings and search results. In Qdrant, use in-memory cache for frequent queries. For LLM, use Redis cache with TTL.

4.2. Error Handling Pipeline

• Retry: on Qdrant or LLM errors (up to 3 attempts).
• Fallback: if search is empty, respond with "I didn't find information in the database."
• Logging: log each step (time, errors, token count).

4.3. Monitoring

Use Prometheus + Grafana for metrics:
- p50/p95 search latency
- cache hit rate
- LLM error count
- average relevance (if manual labeling exists)

4.4. Integration with External Services

If your RAG pipeline needs to process data from CRM or ERP, flexible integration is required. ASI Biont supports connection to Qdrant, Elasticsearch, and other services via API — more details at asibiont.com. This makes it easy to embed RAG into existing infrastructure.

5. Conclusion

A production-ready RAG pipeline is not magic but engineering work. Key points:
- Chunking — use Recursive or Semantic, not fixed.
- Search — definitely hybrid (dense + sparse).
- Reranking — adds +10-15% accuracy.
- Monitoring — without it, you're blind.

Want to dive deeper? The ASI Biont platform offers a full course on building RAG systems for production: from choosing embedding models to implementation with caching and monitoring. Explore chunking strategies, hybrid search, and Graph RAG — all with code and benchmarks.