How to Build a RAG System for Document Search: A Step-by-Step Guide with LangChain and Qdrant

How to Build a RAG System for Document Search: A Step-by-Step Guide with LangChain and Qdrant

Introduction

Imagine you have hundreds of thousands of pages of corporate documents—from technical documentation to legal contracts. Finding the right information is like finding a needle in a haystack. Traditional keyword search often returns hundreds of irrelevant results, forcing you to sift through mountains of text. But what if you could ask a question in natural language and get a precise answer, backed by source references? That's exactly what Retrieval-Augmented Generation (RAG) does—one of the hottest technologies in AI engineering in 2026.

RAG systems combine the power of large language models (LLMs) with the precision of semantic search. Instead of forcing the model to "remember" all data, you give it access to up-to-date information through a vector database. This not only improves answer quality but also addresses the hallucination problem. In this article, we'll walk through building a production-ready RAG pipeline using LangChain and Qdrant—two key tools in the modern AI engineer's stack.

If you want to dive deeper into the architecture of AI products, including RAG and other pipelines, the asibiont.com platform offers a comprehensive course on this topic. But for now—let's get to work.

What is a RAG System and Why Do You Need It?

Retrieval-Augmented Generation is an architecture that adds a step of retrieving relevant information before generating an answer. Instead of the LLM answering "from memory," you first find suitable document fragments in a vector database and then pass them to the model as context. This provides three key advantages:

• Relevance: The model always uses the latest data, no need for frequent retraining.
• Accuracy: Answers contain fewer hallucinations because they rely on sources.
• Transparency: You can show the user where the answer came from, increasing trust.

Without RAG, document search becomes classic full-text search—good for exact matches but useless for synonyms or paraphrasing. RAG works at the semantic level using vector embeddings.

Step 1: Architecture of a RAG Pipeline

Before writing code, it's important to understand the building blocks of the system. Here's a simplified architecture:

In practice, the pipeline consists of two phases: indexing (one-time, or with updates) and inference (per query). In production, it's crucial to configure chunking correctly—too small chunks lose context, too large ones introduce noise.

Step 2: Choosing the Stack—Why LangChain and Qdrant?

As of 2026, LangChain remains the de facto standard for building LLM pipelines. It provides abstractions for working with models, prompts, and vector stores. Qdrant is a vector database written in Rust, offering high performance, flexible filtering, and easy production setup.

Let's compare Qdrant with popular alternatives:

For startups and medium-sized projects, Qdrant offers an ideal balance of performance and cost. LangChain integrates easily with it via the built-in QdrantVectorStore class.

Step 3: Environment Setup

We'll need:

• Python 3.11+
• Qdrant (locally via Docker or a cloud instance)
• LangChain (latest version)
• Embedding model (e.g., text-embedding-3-small from OpenAI or intfloat/multilingual-e5-large)
• LLM for generation (GPT-4o, Claude 3.5, or Llama 3.1)

Install dependencies:

pip install langchain langchain-community langchain-openai qdrant-client pypdf sentence-transformers

Step 4: Document Indexing

The first stage is data preparation. We have PDF documents that need to be split into chunks, converted to vectors, and saved in Qdrant.

Text Chunking

Choosing a chunking strategy is critical. Here are the main approaches:

Example code with RecursiveCharacterTextSplitter:

from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain_community.document_loaders import PyPDFLoader

loader = PyPDFLoader("document.pdf")
documents = loader.load()

text_splitter = RecursiveCharacterTextSplitter(
    chunk_size=1000,
    chunk_overlap=200,
    separators=["\n\n", "\n", " ", ""]
)
chunks = text_splitter.split_documents(documents)
print(f"Created {len(chunks)} chunks")

Creating Embeddings and Loading into Qdrant

Now we need to create a vector representation for each chunk and save it.

from langchain_openai import OpenAIEmbeddings
from langchain_qdrant import QdrantVectorStore
from qdrant_client import QdrantClient
from qdrant_client.http.models import Distance, VectorParams

# Connect to Qdrant
client = QdrantClient(url="http://localhost:6333")  # or use a cloud URL

# Create a collection
collection_name = "my_docs"
client.recreate_collection(
    collection_name=collection_name,
    vectors_config=VectorParams(size=1536, distance=Distance.COSINE),
)

# Initialize the store with LangChain
embeddings = OpenAIEmbeddings(model="text-embedding-3-small")
vector_store = QdrantVectorStore(
    client=client,
    collection_name=collection_name,
    embedding=embeddings,
)

# Add documents
vector_store.add_documents(documents=chunks)
print("Indexing complete")

Note: The vector size (1536) must match the embedding model's dimensions. text-embedding-3-small gives 1536 dimensions. If using other models, check the dimensionality.

Step 5: Search and Generation

The second phase is processing the user's query. We find similar chunks, form the context, and pass it to the LLM.

Semantic Search

# Search for top-3 relevant chunks
query = "What are the return conditions?"
results = vector_store.similarity_search(query, k=3)

for i, doc in enumerate(results):
    print(f"Result {i+1}: {doc.page_content[:200]}...")

Qdrant also supports metadata filtering (payload filtering). For example, you can search only documents from 2026:

from qdrant_client.http.models import Filter, FieldCondition, Range

my_filter = Filter(
    must=[
        FieldCondition(key="year", range=Range(gte=2026))
    ]
)
results = vector_store.similarity_search(query, k=3, filter=my_filter)

Generating an Answer with LangChain

Now let's create a RAG chain using LangChain Expression Language (LCEL):

from langchain_openai import ChatOpenAI
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.runnables import RunnablePassthrough
from langchain_core.output_parsers import StrOutputParser

# Initialize the LLM
llm = ChatOpenAI(model="gpt-4o", temperature=0)

# Prompt template
prompt = ChatPromptTemplate.from_template(
    """You are an assistant answering questions based on documents.
Use ONLY the following context to answer.
If the answer is not in the context, say you don't know.

Context:
{context}

Question: {question}

Answer:"""
)

# Function to format context
def format_docs(docs):
    return "\n\n".join([d.page_content for d in docs])

# RAG chain
rag_chain = (
    {"context": vector_store.as_retriever(search_kwargs={"k": 3}) | format_docs,
     "question": RunnablePassthrough()}
    | prompt
    | llm
    | StrOutputParser()
)

# Execute the query
answer = rag_chain.invoke("What are the return conditions?")
print(answer)

This code forms a production-ready pipeline. as_retriever() returns an object that searches documents and passes them to the prompt.

Step 6: Optimization for Production

Developing a RAG system doesn't end with the basic setup. Here's what to consider for production:

1. Chunking Quality

• Use semantic chunking (e.g., via LangChain's SemanticChunker) to avoid breaking sentences in the middle.
• Experiment with chunk size: 500 to 1500 tokens is a typical range.
• Add metadata: document title, date, page number—this helps with filtering.

2. Search Speed

• Qdrant supports HNSW indexing—enabled by default, but you can tune m and ef_construct parameters.
• Use caching for embeddings of frequent queries.
• Consider hybrid search: a combination of vector and full-text search (Qdrant supports via bm25).

3. Monitoring

• Track latency: target < 1 second for search, < 3 seconds for generation.
• Calculate cost: how many tokens are used for embeddings and generation.
• A/B test different embedding models and LLMs.

4. Security

• Do not pass documents into context that the user doesn't have access to—use payload filtering in Qdrant.
• Set up rate limiting on the API.

Step 7: Monitoring and Cost Optimization

In production, a RAG system can be expensive if costs aren't controlled. Here's a table of typical costs:

To reduce costs:
- Use an open-source embedding model (e.g., BAAI/bge-m3).
- Apply reranking—first search for 20 chunks, then rank top-3, saving LLM tokens.
- Set up a cache for repeated queries.

Conclusion

A RAG system with LangChain and Qdrant is not just a trendy buzzword but a working tool that today allows you to build semantic document search with minimal costs. We've covered the full pipeline: from PDF indexing to answer generation with context. Key takeaways:

• RAG solves the problem of hallucinations and data relevance.
• Qdrant is a powerful and free vector database for self-hosted projects.
• LangChain simplifies integration and allows quick switching between models.
• Production requires optimization of chunking, monitoring, and costs.

Want to learn how to build not only RAG but also AI agents, fine-tune models, and deploy everything on Kubernetes? On asibiont.com, there's an intensive course "Full-Stack AI Engineer" where we cover these topics with production code and practical projects. Join to become an expert in AI engineering.

Start small: take your corporate document, run Qdrant in Docker, and build your first RAG pipeline. The result will surprise you.