Imagine asking a corporate AI assistant a question, and instead of generic phrases, it gives you an exact answer from your internal regulations or technical documentation. This isn't magic, but Retrieval Augmented Generation (RAG) — a technology that allows language models to work with your data, not just public information. In this article, we'll break down how to build a RAG system with your own hands: from document loading to vector search.
RAG solves the main problem of large language models (LLMs): they don't know your specific data. Without RAG, AI can hallucinate or give outdated answers. With RAG, you literally "feed" the model your knowledge base, and it responds based on current facts. Let's see how this works in practice.
What is RAG and Why Do You Need It?
Retrieval Augmented Generation is an architecture that combines retrieval and text generation. The process looks like this:
1. You upload documents (PDF, Word, Markdown, HTML).
2. The system splits them into chunks (chunking) and creates vector representations (embeddings).
3. Upon a query, AI searches for relevant chunks in a vector database.
4. The found fragments are passed to the LLM as context for generating an answer.
The main advantage of RAG is relevance and accuracy. You can update the knowledge base without retraining the model. This is ideal for support chatbots, internal knowledge portals, and document analysis.
Step 1: Loading and Preparing Documents
The first stage is to gather all data sources. These can be:
- Text files (.txt, .md)
- Office documents (.docx, .pptx)
- PDF files
- HTML pages
- Databases (SQL, NoSQL)
For loading, use libraries like langchain (Python) or unstructured. It's important to clean the text of extra characters, tables, and formatting. After cleaning, the data is ready for chunking.
Step 2: Chunking — Splitting into Pieces
Chunking is dividing documents into small fragments. The quality of search depends on the chunk size. Too small chunks lose context, too large ones reduce accuracy.
| Chunking Type | Size | Pros | Cons |
|---|---|---|---|
| Fixed | 256-512 tokens | Simple implementation | Breaks sentences |
| By sentences | 1-3 sentences | Preserves meaning | Many chunks |
| Recursive | 1000 tokens with overlap | Balance of accuracy and context | More complex setup |
I recommend using recursive chunking with an overlap of 10-20%. This preserves text coherence and improves search results.
Step 3: Embeddings — Turning Text into Vectors
Embeddings are numerical representations of text. They allow AI to understand semantic similarity between a query and documents. Popular embedding models:
- OpenAI text-embedding-3-small (paid)
- intfloat/multilingual-e5-small (free, open-source)
- BAAI/bge-base-en-v1.5 (for English)
Each chunk is converted into a vector with a dimension of 384-1536 numbers. These vectors are stored in a vector database.
Step 4: Vector Database — Knowledge Storage
A vector database allows fast searching for semantically similar vectors. Main options:
- Chroma — lightweight, embeddable in code
- Pinecone — cloud-based, scalable
- Qdrant — self-hosted, fast
- Weaviate — supports hybrid search
For a start, use Chroma — it installs with one command and works locally. Example code:
from langchain.vectorstores import Chroma
from langchain.embeddings import HuggingFaceEmbeddings
embeddings = HuggingFaceEmbeddings(model_name="intfloat/multilingual-e5-small")
vectorstore = Chroma.from_documents(documents, embeddings)
Step 5: Vector Search and Answer Generation
When a user asks a question, the system:
1. Converts the query into an embedding using the same model.
2. Searches for the top-K (usually 3-5) nearest chunks in the vector database.
3. Passes the found fragments to the LLM along with instructions.
Example prompt:
"Answer the question based on the provided context."
Comments