You ask your shiny new RAG assistant a simple question — "What's our refund policy for annual plans?" — and it confidently replies with a paragraph about shipping times. Or worse, it invents a policy that doesn't exist. You check the logs: the vector database returned five chunks, they all look vaguely related, and the LLM did what LLMs do — turned vague context into a fluent, wrong answer. For a deeper dive into this topic, see Stop Asking LLMs to Guess CVEs.
The problem is almost never the LLM. It's retrieval. And the fixes are concrete: chunk sizes, hybrid search, metadata filters, reranking, and a way to measure recall before you blame the model. This article walks through the real mechanics, with a working pgvector/Qdrant example and before/after numbers you can reproduce.
Why vector search feels like magic until it doesn't
A vector database stores embeddings — dense numeric representations of text, images, or audio — and finds nearest neighbours by cosine or dot-product similarity. It's brilliant at fuzzy meaning: "how do I cancel my subscription" matches "terminating your plan" even with zero shared words.
But that same fuzziness is the trap. Pure embedding search has no concept of exact tokens. If your corpus contains product codes like SKU-4471 or legal clause numbers like Section 12.3(b), the embedding of SKU-4471 is nearly identical to SKU-4472 — one digit apart in text, almost the same point in vector space. The search returns the wrong SKU with high confidence, the LLM repeats it, and your support team gets a ticket. For a deeper dive into this topic, see A/B Testing That Survives Scrutiny.
A walkthrough: fixing a broken RAG pipeline step by step
Imagine a B2B SaaS company with 4,000 support articles, an internal wiki, and a RAG bot for the customer success team. The bot answers 70% of questions well and 30% badly — the classic ratio. Here's how a practitioner diagnoses and fixes it, in order.
Step 1: measure retrieval recall before touching the prompt
Before changing anything, build a small evaluation set: 50 real questions with the correct source chunk marked. Then compute recall@5 — the share of questions where the correct chunk appears in the top 5 results. If recall@5 is below 0.8, no prompt engineering will save you. In the example above, recall@5 was 0.61. The LLM was being fed irrelevant context 4 times out of 10.
Step 2: fix chunking strategy
Most teams start with 1,000-token chunks and 0 overlap. That's too big: a chunk covering three topics embeds as a blurry average, and the retriever pulls it for any of the three. The fix is smaller, overlapping chunks — typically 300–500 tokens with 10–15% overlap (30–75 tokens). Overlap prevents a sentence that straddles a boundary from being split in half.
Better still: chunk by structure. Split on headings first, then on paragraphs, and only then by token count. A support article about refunds should produce one chunk per policy clause, not one chunk for the whole page.
Step 3: add hybrid search (BM25 + vector)
Pure embeddings miss exact terms. BM25 — the classic keyword ranking function — catches them. Hybrid search runs both and fuses the scores, usually with Reciprocal Rank Fusion (RRF): each result gets a score of 1/(k + rank), where k is typically 60, and the scores are summed.
In the walkthrough, adding BM25 fixed the SKU-4471 problem and lifted recall@5 from 0.74 to 0.86. That's the single highest-leverage change in most RAG pipelines.
Step 4: metadata filtering and partition keys
If your bot serves multiple tenants, a query from tenant A must never retrieve tenant B's chunks. The fix is a metadata filter on tenant_id — or, in Qdrant, a partition key on that field, which physically isolates data for faster filtered search. In pgvector, you add a WHERE tenant_id = $1 clause alongside the vector order-by.
Step 5: rerank the top 20
A cross-encoder reranker (e.g., a small BERT-based model) scores each query–chunk pair jointly and is far more accurate than cosine similarity — but too slow to run over the whole corpus. The pattern: retrieve top 20 with hybrid search, rerank to top 5, feed those to the LLM. In the walkthrough, reranking lifted recall@5 to 0.93 and — just as important — pushed the correct chunk to rank 1 in most cases, which reduces hallucination because the LLM sees the right context first.
Step 6: a working pgvector snippet
Here's the core of the hybrid retrieval in PostgreSQL with pgvector:
-- vector channel
SELECT id, content, 1 - (embedding <=> $1) AS score
FROM chunks
WHERE tenant_id = $2
ORDER BY embedding <=> $1
LIMIT 20;
-- lexical channel (PostgreSQL full-text search as a stand-in for BM25)
SELECT id, content, ts_rank(tsv, plainto_tsquery($3)) AS score
FROM chunks
WHERE tenant_id = $2 AND tsv @@ plainto_tsquery($3)
ORDER BY score DESC
LIMIT 20;
Then fuse the two result sets in application code with RRF. In Qdrant, the same idea is a single query with prefetch for each channel and fusion: "rrf".
Where the scheme itself breaks
Even a well-tuned pipeline has failure modes that no amount of chunking will fix. Spot them early.
1. The query is ambiguous. "How do I cancel?" could mean a subscription, an order, or a meeting. Embedding search picks one interpretation and confidently returns it. The fix is query rewriting: ask the LLM to expand the query into two or three variants, retrieve for each, and merge. Cost: extra latency and tokens.
2. The answer spans multiple chunks. A policy that says "refunds are available within 30 days, except for annual plans, which are non-refundable after 14 days" may be split across two chunks. Retrieval returns only one, and the LLM answers half the truth. The fix is parent-document retrieval: embed small chunks but return the larger parent section to the LLM. Trade-off: more context tokens, higher cost per query.
3. The corpus drifts. New articles are added, old ones edited, but the embeddings are stale. A chunk that was accurate six months ago now contradicts the current policy. The fix is incremental re-indexing triggered by content changes, plus a periodic full re-embed when you upgrade the embedding model — because vectors from different models are not comparable.
4. The embedding model doesn't fit the domain. A general-purpose model may not distinguish "SLA breach" from "SLO miss" if your corpus is full of jargon. The fix is either a domain-tuned model or, more cheaply, adding a keyword channel (which you already have with BM25).
Learning paths and their real trade-offs
You can learn this by building, by reading, or by a structured course. Each has a cost.
Self-study is cheapest in money and most expensive in time. You'll wire up pgvector from a blog post in an afternoon, but you won't know why recall is 0.6 — you'll tweak prompts for a week before suspecting chunking. The trap is that RAG failures look like LLM failures.
Building a side project is the best teacher if you have a real corpus and real questions. Take your company wiki, build the bot, and measure recall@5 on 50 questions. The cost is the hours you spend debugging alone, with no one to check your evaluation set for bias.
A structured course trades money for a map. The value is in the comparisons — pgvector vs Qdrant vs Pinecone, HNSW vs IVF vs DiskANN, when hybrid search is worth the complexity and when it isn't — and in having someone check your decisions. If you want that map, Vector Databases — From Search to RAG walks through exactly these trade-offs with working code and comparison tables; the lessons are personalised and adapt to your level. For a broader view of how LLMs fail on technical tasks, see our article on why LLMs guess CVEs and how to prompt them for real work.
Who this path suits — and who it doesn't
This work suits engineers who like debugging systems, not just models. If you enjoy building evaluation sets, reading recall curves, and arguing about chunk sizes, you'll thrive. It also suits data engineers moving into AI infrastructure, because the skills — indexing, filtering, scaling — are familiar.
It does not suit people who want to "just use the API." Managed vector databases are easy to start and hard to tune; the tuning is where the value is, and it requires patience with numbers. It also doesn't suit those who expect a single tool to solve retrieval — hybrid search, reranking, and evaluation are all necessary, and none is optional in production.
If you've ever watched a RAG bot confidently answer the wrong question, you already know why this matters. The fix isn't a better prompt — it's a better retrieval pipeline. Start by measuring recall@5 on your own corpus, then work through chunking, hybrid search, and reranking in that order. For a structured path through the trade-offs, Vector Databases — From Search to RAG covers the full pipeline with working examples and feedback on your decisions.