RAG Prompts Playbook: Indexing, Hybrid Search, and Generation with Vector Databases

RAG Prompts Playbook: Indexing, Hybrid Search, and Generation with Vector Databases

Retrieval-Augmented Generation (RAG) has become the default architecture for production LLM applications. By mid-2026, the majority of enterprise AI deployments rely on RAG-style pipelines backed by vector databases such as Pinecone, Qdrant, Weaviate, and Milvus. Yet most teams still treat prompts as an afterthought — a single "answer the question using the context" instruction glued to the top of a retrieval loop.

That is a mistake. Prompts are not just a generation-layer concern. In a well-designed RAG system, prompts govern how documents are chunked, how embeddings are generated, how queries are rewritten for hybrid search, and how retrieved chunks are synthesized into grounded answers. This article is a practical collection of ready-to-use prompts for each stage of the RAG pipeline: indexing, search, and generation. Use it as a cheat sheet for your next vector-database-backed project.

The Three-Stage RAG Pipeline and Where Prompts Fit

Before diving into prompts, it helps to visualize the pipeline. RAG has three distinct stages, and each one contains at least one promptable component:

This table is a mental map. Whenever you debug a bad RAG answer, ask yourself which stage produced the bad input — the prompts below will help you isolate and fix the problem.

Stage 1: Indexing — Prompts for Smarter Chunking

Retrieval quality is set at indexing time. If chunks are too large, they contain passages that dilute the embedding; if too small, they lose semantic context. Prompt-based semantic chunking fixes arbitrary chunk sizes.

Prompt 1.1: Semantic Chunking

You are a document segmentation specialist. Split the input text below into semantically cohesive chunks of 200–400 tokens.
Rules:
- Do not split a paragraph unless it exceeds 400 tokens.
- Keep code blocks, tables, and lists intact.
- Start a new chunk only when a new idea or subtopic begins.
Output JSON with the field "chunks": an array of strings.
Input:
{{document_text}}

This prompt does not require an LLM call per thousand chunks — run it once per document or chapter. The output feeds directly into an embedding model and then into the vector database.

Prompt 1.2: Metadata Enrichment

Metadata is the underrated half of hybrid search. A chunk without metadata is blind; a chunk with title, section, date, and entity tags is addressable.

Assign metadata for each chunk I provide. Return JSON with keys: "title", "section", "doc_type", "entities" (a list of proper nouns), "language", "date_published", and "summary" (one sentence).
Do not invent values: if an attribute is unknown, set it to null.
Chunk: {{chunk_text}}

The resulting metadata becomes filterable fields in Qdrant or Milvus, turning a pure vector lookup into filtered search.

Stage 2: Retrieval — Prompts for Query Understanding and Hybrid Search

With the index ready, the next bottleneck is the query. Users type vague, multi-intent, or domain-specific questions — and the vector database returns whatever embedding similarity dictates. Query rewriting fixes that.

Prompt 2.1: Query Rewriting for Hybrid Search

Hybrid search combines dense vectors with keyword (BM25) matching. A raw user query like "how does the new pricing work for API users?" confuses BM25 with stopwords. Rewrite it into two variants.

Rewrite the user query to improve hybrid retrieval (dense + BM25). Requirements:
- Return exactly two strings: "dense_query" and "kw_query".
- dense_query: preserve user intent, expand abbreviations, 1–2 sentences.
- kw_query: a keyword-heavy phrase for lexical matching; no parentheses or boolean operators.
User query: {{user_query}}
Output as JSON.

A concrete example: the query above becomes dense_query = "how are API users charged under the new pricing plan?" and kw_query = "API pricing plan charges". These two strings feed into the dual retrieval paths of your vector database.

Prompt 2.2: Multi-Query Decomposition

Roughly 30% of real queries contain two combined sub-questions. One embedding cannot represent two topics, so recall collapses.

Decompose the user query into independent sub-queries that can be searched separately.
Requirements:
- Each sub-query must be self-contained (no pronouns).
- If the query is atomic, return it unchanged.
- Output JSON: {"sub_queries": ["..."]}
Query: {{user_query}}

Issued in parallel, the union of results is re-ranked. This single prompt usually raises recall by 20–30% in question-answering benchmarks.

Stage 3: Generation — Prompts for Grounded Answers

Retrieval only finds candidates. The generation prompt decides whether the final output is faithful, concise, and properly cited. Two workhorse prompts below.

Prompt 3.1: Grounded Generation with Mandatory Attribution

RAG answers that hallucinate facts not present in the context have two causes: a weak retriever and a permissive generation prompt. The second is easy to fix.

You are a bilingual analyst assistant with access to retrieved context only. Answer the question using ONLY the provided context. If the context lacks the information, say: "The retrieved documents do not contain this information."
Every factual claim must be followed by a citation like [src:N] where N is the source index of the context chunk.
Answer in the user's language, in 4–8 sentences, structured as a concise paragraph.
Context chunks:
{{context_chunks}}
Question: {{question}}

The [src:N] convention is important: it lets you compute faithfulness programmatically by checking whether each cited fact actually appears in chunk N.

Prompt 3.2: Contrastive Refusal and Fallback

Sometimes the correct answer is "I don't know". A refusal prompt prevents the assistant from inventing a blend of retrieved and parametric knowledge.

Use only {{context_chunks}} to answer {{question}}.
- If the context partially answers the question, answer that part and explicitly list missing aspects.
- If the context is irrelevant, respond with: "The knowledge base has no relevant information for this request." and suggest the closest related topic if one exists.
- Never inject outside knowledge, even if the question seems simple (e.g. capital cities, formulas).

The last line is a deliberate trap: it turns the model's parametric knowledge from a liability into a controlled fallback.

Evaluation Prompts: Closing the Loop

Prompts are not only for the runtime pipeline. They can also score retrieval and generation quality, replacing manual inspection. Use an LLM-as-a-judge prompt with explicit rubrics:

You are an RAG evaluator. Score the candidate answer on 0–5 for each criterion, then output JSON.
Criteria:
- faithfulness (0–5): each claim is attributable to the context.
- relevance (0–5): the answer matches the question, not just the context.
- conciseness (0–5): no fluff or repetition.
- citation correctness (0–5): [src:N] markers point to matching chunks.
Context: {{context}}
Question: {{question}}
Answer: {{answer}}

Run this over 100–200 offline examples and you will get a stable metric — call it the RAG-QA score — that you can track after every pipeline change.

Prompt Selection: A Cheat Sheet

The table below summarizes which prompt to use for each problem you are likely to hit.

Recommendations for Production RAG

Takeaways from running vector databases in production:

First, treat prompts as versioned artifacts. Store each prompt in the same repository as your code, with a version hash — prompt changes are behavior changes. Second, prefer deterministic post-processing over prompt trickery for citation parsing: the judge prompt gives a score, but parsing [src:N] with a regex is more reliable than asking the model for a citation tree. Third, run the semantic chunking prompt once per document at indexing time, but run query rewriting on every request — the former is latency-tolerant, the latter is not.

Fourth, monitor each stage separately. Index quality is measured by chunk density and metadata coverage; retrieval quality by recall@k on a labeled eval set; generation quality by the faithfulness score from the judge prompt. Do not let one end-to-end metric hide a broken stage.

Conclusion

RAG is not "embed a document and hope for the best". Each of the three pipeline stages — indexing, retrieval, and generation — has a prompt boundary where quality is won or lost. Semantic chunking prompts keep your vector database clean; query rewriting and decomposition prompts make hybrid search actually hybrid; grounded generation prompts with mandatory citations keep answers honest; and judge prompts turn quality into a number you can improve.

Start with the two highest-leverage changes: the semantic chunking prompt at indexing time and grounded generation with [src:N] citations. Roll them into your pipeline, measure the faithfulness score, and iterate on the remaining prompts.

Now go improve your RAG.