How to Build a RAG System for Document Search: A Step-by-Step Guide with LangChain and Qdrant

## How to Build a RAG System for Document Search: A Step-by-Step Guide with LangChain and Qdrant ### Introduction Imagine you have hundreds of thousands of pages of corporate documents—from technical documentation to legal contracts. Finding the right information is like finding a needle in a haystack. Traditional keyword search often returns hundreds of irrelevant results, forcing you to sift through mountains of text. But what if you could ask a question in natural language and get a precise answer, backed by source references? That's exactly what Retrieval-Augmented Generation (RAG) does—one of the hottest technologies in AI engineering in 2026. RAG systems combine the power of large language models (LLMs) with the precision of semantic search. Instead of forcing the model to "remember" all data, you give it access to up-to-date information through a vector database. This not only improves answer quality but also addresses the hallucination problem. In this article, we'll walk through building a production-ready RAG pipeline using LangChain and Qdrant—two key tools in the modern AI engineer's stack. If you want to dive deeper into the architecture of AI products, including RAG and other pipelines, the asibiont.com platform offers a comprehensive course on this topic. But for now—let's get to work. ### What is a RAG System and Why Do You Need It? Retrieval-Augmented Generation is an architecture that adds a step of retrieving relevant information before generating an answer. Instead of the LLM answering "from memory," you first find suitable document fragments in a vector database and then pass them to the model as context. This provides three key advantages: - **Relevance**: The model always uses the latest data, no need for frequent retraining. - **Accuracy**: Answers contain fewer hallucinations because they rely on sources. - **Transparency**: You can show the user where the answer came from, increasing trust. Without RAG, document search becomes classic full-text search—good for exact matches but useless for synonyms or paraphrasing. RAG works at the semantic level using vector embeddings. ### Step 1: Architecture of a RAG Pipeline Before writing code, it's important to understand the building blocks of the system. Here's a simplified architecture: | Component | Task | Example Tool | |-----------|------|--------------| | Document Index | Split texts into chunks, create embeddings, load into vector DB | LangChain, Qdrant, Sentence-Transformers | | Search Module | Find top-K relevant chunks for a query | Qdrant (semantic search) | | Context Prompt | Assemble found chunks and pass them to the LLM | LangChain Prompt Template | | Answer Generator | Formulate an answer based on context | OpenAI GPT-4o, Llama 3, Claude | | Monitoring | Track latency, cost, quality | LangSmith, Grafana | In practice, the pipeline consists of two phases: indexing (one-time, or with updates) and inference (per query). In production, it's crucial to configure chunking correctly—too small chunks lose context, too large ones introduce noise. ### Step 2: Choosing the Stack—Why LangChain and Qdrant? As of 2026, LangChain remains the de facto standard for building LLM pipelines. It provides abstractions for working with models, prompts, and vector stores. Qdrant is a vector database written in Rust, offering high performance, flexible filtering, and easy production setup. Let's compare Qdrant with popular alternatives: | Feature | Qdrant | Chroma | Pinecone | |---------|--------|--------|----------| | Hosting Type | Self-hosted / Cloud | Self-hosted | Cloud-only | | Language | Rust (fast) | Python | Closed-source | | Filtering | Advanced (payload filtering) | Basic | Advanced | | Price | Free (self-hosted) | Free | Paid (per vector count) | | Multi-vector Support | Yes | No | Yes | For startups and medium-sized projects, Qdrant offers an ideal balance of performance and cost. LangChain integrates easily with it via the built-in `QdrantVectorStore` class. ### Step 3: Environment Setup We'll need: - Python 3.11+ - Qdrant (locally via Docker or a cloud instance) - LangChain (latest version) - Embedding model (e.g., `text-embedding-3-small` from OpenAI or `intfloat/multilingual-e5-large`) - LLM for generation (GPT-4o, Claude 3.5, or Llama 3.1) Install dependencies: ```bash pip install langchain langchain-community langchain-openai qdrant-client pypdf sentence-transformers ``` ### Step 4: Document Indexing The first stage is data preparation. We have PDF documents that need to be split into chunks, converted to vectors, and saved in Qdrant. #### Text Chunking Choosing a chunking strategy is critical. Here are the main approaches: | Strategy | Description | When to Use | |----------|-------------|-------------| | RecursiveCharacterTextSplitter | Recursively splits by separators (paragraphs, sentences) | Universal | | SemanticChunker | Splits by meaning using embeddings | When semantic integrity matters | | TokenTextSplitter | Splits by tokens (for models with limits) | For precise length control | Example code with RecursiveCharacterTextSplitter: ```python from langchain.text_splitter import RecursiveCharacterTextSplitter from langchain_community.document_loaders import PyPDFLoader loader = PyPDFLoader("document.pdf") documents = loader.load() text_splitter = RecursiveCharacterTextSplitter( chunk_size=1000, chunk_overlap=200, separators=["\n\n", "\n", " ", ""] ) chunks = text_splitter.split_documents(documents) print(f"Created {len(chunks)} chunks") ``` #### Creating Embeddings and Loading into Qdrant Now we need to create a vector representation for each chunk and save it. ```python from langchain_openai import OpenAIEmbeddings from langchain_qdrant import QdrantVectorStore from qdrant_client import QdrantClient from qdrant_client.http.models import Distance, VectorParams # Connect to Qdrant client = QdrantClient(url="http://localhost:6333") # or use a cloud URL # Create a collection collection_name = "my_docs" client.recreate_collection( collection_name=collection_name, vectors_config=VectorParams(size=1536, distance=Distance.COSINE), ) # Initialize the store with LangChain embeddings = OpenAIEmbeddings(model="text-embedding-3-small") vector_store = QdrantVectorStore( client=client, collection_name=collection_name, embedding=embeddings, ) # Add documents vector_store.add_documents(documents=chunks) print("Indexing complete") ``` Note: The vector size (1536) must match the embedding model's dimensions. `text-embedding-3-small` gives 1536 dimensions. If using other models, check the dimensionality. ### Step 5: Search and Generation The second phase is processing the user's query. We find similar chunks, form the context, and pass it to the LLM. #### Semantic Search ```python # Search for top-3 relevant chunks query = "What are the return conditions?" results = vector_store.similarity_search(query, k=3) for i, doc in enumerate(results): print(f"Result {i+1}: {doc.page_content[:200]}...") ``` Qdrant also supports metadata filtering (payload filtering). For example, you can search only documents from 2026: ```python from qdrant_client.http.models import Filter, FieldCondition, Range my_filter = Filter( must=[ FieldCondition(key="year", range=Range(gte=2026)) ] ) results = vector_store.similarity_search(query, k=3, filter=my_filter) ``` #### Generating an Answer with LangChain Now let's create a RAG chain using LangChain Expression Language (LCEL): ```python from langchain_openai import ChatOpenAI from langchain_core.prompts import ChatPromptTemplate from langchain_core.runnables import RunnablePassthrough from langchain_core.output_parsers import StrOutputParser # Initialize the LLM llm = ChatOpenAI(model="gpt-4o", temperature=0) # Prompt template prompt = ChatPromptTemplate.from_template( """You are an assistant answering questions based on documents. Use ONLY the following context to answer. If the answer is not in the context, say you don't know. Context: {context} Question: {question} Answer:""" ) # Function to format context def format_docs(docs): return "\n\n".join([d.page_content for d in docs]) # RAG chain rag_chain = ( {"context": vector_store.as_retriever(search_kwargs={"k": 3}) | format_docs, "question": RunnablePassthrough()} | prompt | llm | StrOutputParser() ) # Execute the query answer = rag_chain.invoke("What are the return conditions?") print(answer) ``` This code forms a production-ready pipeline. `as_retriever()` returns an object that searches documents and passes them to the prompt. ### Step 6: Optimization for Production Developing a RAG system doesn't end with the basic setup. Here's what to consider for production: #### 1. Chunking Quality - Use semantic chunking (e.g., via LangChain's `SemanticChunker`) to avoid breaking sentences in the middle. - Experiment with chunk size: 500 to 1500 tokens is a typical range. - Add metadata: document title, date, page number—this helps with filtering. #### 2. Search Speed - Qdrant supports HNSW indexing—enabled by default, but you can tune `m` and `ef_construct` parameters. - Use caching for embeddings of frequent queries. - Consider hybrid search: a combination of vector and full-text search (Qdrant supports via `bm25`). #### 3. Monitoring - Track latency: target < 1 second for search, < 3 seconds for generation. - Calculate cost: how many tokens are used for embeddings and generation. - A/B test different embedding models and LLMs. #### 4. Security - Do not pass documents into context that the user doesn't have access to—use payload filtering in Qdrant. - Set up rate limiting on the API. ### Step 7: Monitoring and Cost Optimization In production, a RAG system can be expensive if costs aren't controlled. Here's a table of typical costs: | Component | Cost (per 1000 queries) | How to Optimize | |-----------|-------------------------|-----------------| | Embeddings (OpenAI) | $0.13 | Use local models (Sentence-Transformers) | | LLM (GPT-4o) | $2.50 (input) + $10.00 (output) | Use smaller models, caching, shorten context | | Qdrant (self-hosted) | $0 (only hardware) | Optimize indexes, reduce chunk count | | Infrastructure | $20–100/month | Kubernetes for scaling | To reduce costs: - Use an open-source embedding model (e.g., `BAAI/bge-m3`). - Apply reranking—first search for 20 chunks, then rank top-3, saving LLM tokens. - Set up a cache for repeated queries. ### Conclusion A RAG system with LangChain and Qdrant is not just a trendy buzzword but a working tool that today allows you to build semantic document search with minimal costs. We've covered the full pipeline: from PDF indexing to answer generation with context. Key takeaways: - RAG solves the problem of hallucinations and data relevance. - Qdrant is a powerful and free vector database for self-hosted projects. - LangChain simplifies integration and allows quick switching between models. - Production requires optimization of chunking, monitoring, and costs. Want to learn how to build not only RAG but also AI agents, fine-tune models, and deploy everything on Kubernetes? On asibiont.com, there's an intensive course "Full-Stack AI Engineer" where we cover these topics with production code and practical projects. Join to become an expert in AI engineering. Start small: take your corporate document, run Qdrant in Docker, and build your first RAG pipeline. The result will surprise you.

How to Build a RAG System for Document Search: A Step-by-Step Guide with LangChain and Qdrant

Comments

Recent articles

How to Automate Production System Monitoring with Prometheus and Grafana: A Step-by-Step Guide

5 Key QA Automation Trends in 2026: AI Agents, ROI, and the Playwright vs Selenium Shift

How to Automate Landing Page A/B Testing with AI Agents and GPT Funnels: A Step-by-Step Guide

Multimodal AI in 2026: How Vision, Audio, and Video Are Rewriting Business Rules

Why Your Personal Website Needs JSON-LD Right Now (And How AI Made It Simple)

How to Automate Manual Data Entry into CRM with AI Agents: A Step-by-Step Guide 2026

From 10 Days to 6: How a Fintech Company Automated IFRS 9 Reporting and Cut Close Time by 40%

How to Build a RAG System in Python: From Chunking to Hybrid Search in Production

5 Predictions for Data Engineering in 2026: From Streaming to AI-Driven Pipelines