15 Prompts for RAG Systems: Indexing, Search, and Generation

Introduction

Retrieval-Augmented Generation (RAG) has become the backbone of modern AI applications that need to answer questions based on private or dynamic knowledge bases. But building a high-quality RAG pipeline is not just about plugging a vector database into a large language model (LLM). The real magic—and the biggest pain points—lie in three areas: indexing (how you chunk and embed documents), retrieval (how you search and re-rank results), and generation (how you prompt the LLM to use retrieved context).

This article presents a curated collection of 15 practical prompts that address common RAG challenges. Each prompt includes a task description, the exact prompt text, and a realistic example output. You can adapt these prompts to your own vector store, embedding model, and LLM. Whether you are building a customer support bot, a research assistant, or an internal knowledge base, these prompts will help you debug, optimize, and scale your RAG system.

Indexing Prompts

1. Semantic Chunking Strategy

• Task: Given a long document (e.g., a whitepaper or legal contract), generate a set of chunk boundaries that preserve semantic units.
• Prompt:
  Document: {full_text} Task: Split this document into coherent chunks. Each chunk should contain a complete idea, paragraph, or section. Avoid splitting mid-sentence or mid-table. Output the chunks as a JSON array with fields "chunk_id", "start_char", "end_char", and "summary".
• Example Result:
  json [ { "chunk_id": 1, "start_char": 0, "end_char": 1500, "summary": "Introduction to RAG and its benefits over pure LLM generation." }, { "chunk_id": 2, "start_char": 1501, "end_char": 3200, "summary": "Architecture: indexing pipeline, embedding models, and vector database choices." } ]

2. Metadata Extraction for Chunks

• Task: For each chunk, extract metadata like document title, section heading, date, and entity mentions.
• Prompt:
  ```
  Chunk text: {chunk_text}
  Extract the following metadata:
• Source document title (if present)
• Section heading (e.g., "3.2. Evaluation")
• Date mentioned (ISO format)
• Top 3 named entities (person, organization, location)
• Keywords (up to 5)
  Return as JSON.
  ```
• Example Result:
  json { "source_title": "RAG Best Practices", "section_heading": "Chunking Strategies", "date": "2026-03-15", "entities": ["OpenAI", "Pinecone", "San Francisco"], "keywords": ["chunk size", "overlap", "semantic boundaries"] }

3. Embedding Quality Check

• Task: Evaluate whether the embedding model captures semantic similarity correctly for a set of chunk pairs.
• Prompt:
  ```
  Pairs of chunks:
  A: "The cat sat on the mat."
  B: "A feline rested on a rug."
  C: "The stock market crashed."
  D: "The cat sat on the mat."

For each pair (A,B), (A,C), (A,D), assign a similarity score from 0 (completely unrelated) to 1 (identical). Explain your reasoning. Then suggest whether the embedding model likely preserves semantic meaning for this domain.
- **Example Result:**
(A,B): 0.85 — synonyms (cat→feline, mat→rug) but slightly different structure.
(A,C): 0.05 — completely unrelated topics.
(A,D): 1.0 — exact duplicate.
The embedding model appears to capture semantic similarity well for simple sentence pairs. For domain-specific terms (e.g., legal jargon), test with real documents.
```

4. Chunk Size Optimization

• Task: Given a sample document and a target chunk size (e.g., 512 tokens), suggest an optimal overlap percentage and justify.
• Prompt:
  ```
  Document excerpt: {excerpt}
  Target chunk size: 512 tokens
  Current overlap: 10%

Evaluate the trade-offs: small chunks improve precision but lose context; large chunks improve recall but may add noise. Recommend an overlap percentage (10%, 20%, or 30%) and explain why for this specific document type (e.g., technical manual, conversational transcript).
- **Example Result:**
I recommend 20% overlap for this technical manual. Technical content often has cross-references between sections, and 20% overlap ensures that key terms (e.g., "API endpoint") appear in at least two chunks, improving retrieval recall without doubling storage cost.
```

Retrieval Prompts

5. Hybrid Search Query Rewriting

• Task: Convert a user query into a form that works well for both dense (vector) and sparse (BM25) search.
• Prompt:
  ```
  Original query: "How to reset my password if I forgot it?"

Rewrite this query into:
1. A dense vector query (semantically expanded, natural language)
2. A sparse keyword query (important terms, synonyms)

Output as JSON with fields "dense_query" and "sparse_query".
- **Example Result:**json
{
"dense_query": "Procedure for password reset when user cannot remember current password",
"sparse_query": "password reset forgot recovery"
}
```

6. Re-Ranking with Cross-Encoder

• Task: Given a query and a list of candidate chunks, re-rank them using a cross-encoder model.
• Prompt:
  ```
  Query: "How do I configure SSL for my web server?"
  Candidates:
• "SSL certificates can be purchased from a CA. Install them on your server."
• "Web server configuration involves editing the httpd.conf file."
• "To configure SSL, generate a private key and CSR, then install the certificate."

Re-rank the candidates from most relevant to least relevant. For each, give a relevance score (0-1) and a one-sentence justification.
- **Example Result:**
1. Score: 0.92 — "To configure SSL, generate a private key and CSR, then install the certificate." (directly addresses the query)
2. Score: 0.45 — "SSL certificates can be purchased from a CA. Install them on your server." (partially relevant, but vague)
3. Score: 0.10 — "Web server configuration involves editing the httpd.conf file." (too general, no SSL mention)
```

7. Query Decomposition for Multi-Hop QA

• Task: Break a complex question into simpler sub-questions that can be answered from separate chunks.
• Prompt:
  ```
  Complex question: "What was the revenue of the company that acquired XYZ Corp in 2025?"

Decompose into sub-questions that retrieve:
1. The company that acquired XYZ Corp in 2025.
2. The revenue of that company for the relevant fiscal year.

For each sub-question, suggest a possible chunk topic (e.g., "M&A announcements", "annual report").
- **Example Result:**json
[
{
"sub_question": "Which company acquired XYZ Corp in 2025?",
"chunk_topic": "Mergers and acquisitions press release"
},
{
"sub_question": "What was the 2025 annual revenue of the acquiring company?",
"chunk_topic": "Annual financial statement"
}
]
```

8. Filter-Prioritized Retrieval

• Task: Given a query and metadata filters (e.g., date range, category), retrieve top-k chunks while respecting filters.
• Prompt:
  ```
  Query: "Latest updates on product pricing"
  Filters: date > 2026-01-01, category = "pricing"

Simulate a retrieval from a vector database. Return the top 3 chunks that satisfy the filters. For each chunk, show its metadata (date, category) and a snippet.
- **Example Result:**
1. Metadata: date=2026-06-15, category=pricing | Snippet: "New pricing tiers effective July 1: Basic $10, Pro $25..."
2. Metadata: date=2026-05-20, category=pricing | Snippet: "Seasonal discount: 20% off annual plans until June 30."
3. Metadata: date=2026-03-01, category=pricing | Snippet: "Price adjustment for enterprise: custom quotes available."
```

Generation Prompts

9. Faithful Answer Generation with Citations

• Task: Generate an answer based on retrieved chunks, with explicit citations to chunk IDs.
• Prompt:
  ```
  Context:
  [Chunk 1]: "RAG reduces hallucination by grounding generation in retrieved documents."
  [Chunk 2]: "Studies show that RAG improves factual accuracy by 30% compared to vanilla LLMs."

Query: "How does RAG reduce hallucination?"

Answer using only the context. For each claim, add a citation in brackets, e.g., [1]. If the context does not contain the answer, say "The provided context does not contain this information."
- **Example Result:**
RAG reduces hallucination by grounding the generation in retrieved documents [1]. Studies show that this approach improves factual accuracy by 30% compared to vanilla LLMs [2].
```

10. Conditional Answering with Confidence

• Task: Answer based on retrieved chunks, but also indicate if the answer is uncertain or if the context is contradictory.
• Prompt:
  ```
  Context:
  [Chunk 1]: "The product launch is scheduled for Q3 2026."
  [Chunk 2]: "The product launch has been postponed to Q1 2027."

Query: "When is the product launch?"

If the context is contradictory, explain the discrepancy. If the context is insufficient, say so. Otherwise, answer concisely.
- **Example Result:**
The context contains contradictory information: Chunk 1 states Q3 2026, while Chunk 2 states Q1 2027. This may be due to a later update. I recommend consulting the most recent source.
```

11. Multi-Turn Conversation with Context

• Task: Maintain a conversation history and answer based on retrieved chunks, using the history to refine retrieval.
• Prompt:
  ```
  History:
  User: "What is RAG?"
  Assistant: "RAG stands for Retrieval-Augmented Generation..."
  User: "How does it handle real-time data?"

Retrieved chunks for this turn:
[Chunk 1]: "RAG can incorporate real-time data by indexing new documents on the fly."

Generate the next assistant response. Use the history to avoid repeating information and to provide a coherent follow-up.
- **Example Result:**
RAG handles real-time data by indexing new documents as soon as they are available, then retrieving them during generation. This allows the model to answer questions about very recent events without retraining.
```

12. Summarization with Source Attribution

• Task: Summarize a set of retrieved chunks into a coherent paragraph, citing each source.
• Prompt:
  ```
  Retrieved chunks:
  [Chunk 1]: "Chunking strategies affect retrieval precision."
  [Chunk 2]: "Overlap between chunks helps maintain context."
  [Chunk 3]: "Metadata tagging improves filter-based search."

Summarize these chunks into 2-3 sentences. Use inline citations like (Chunk 1).
- **Example Result:**
Effective RAG indexing requires careful chunking strategies to balance precision and context (Chunk 1). Using overlap between chunks helps preserve continuity (Chunk 2), while metadata tagging enhances filter-based retrieval (Chunk 3).
```

13. Handling Out-of-Scope Queries

• Task: Detect if a query is outside the knowledge base and respond gracefully.
• Prompt:
  ```
  Context: [No relevant chunks retrieved]
  Query: "What is the meaning of life?"

If the context is empty or irrelevant, respond with: "I cannot answer that based on the available documents. Please ask a question related to the indexed knowledge base."
- **Example Result:**
I cannot answer that based on the available documents. Please ask a question related to the indexed knowledge base.
```

14. Structured Output from Unstructured Context

• Task: Extract structured data (e.g., a table) from retrieved chunks.
• Prompt:
  ```
  Context:
  "Apple reported Q2 revenue of $90 billion. Microsoft reported $62 billion. Google reported $80 billion."

Extract the data into a Markdown table with columns: Company, Revenue (USD), Quarter.
- **Example Result:**

15. Evaluation Prompt for RAG Output

• Task: Evaluate the quality of a RAG-generated answer against the retrieved context.
• Prompt:
  ```
  Query: "What are the benefits of RAG?"
  Context: [Chunk 1]: "RAG reduces hallucination." [Chunk 2]: "RAG improves factual accuracy."
  Generated answer: "RAG reduces hallucination and improves factual accuracy. It also speeds up inference."

Evaluate the answer on:
1. Faithfulness (does it stay within context?)
2. Completeness (does it cover all points in context?)
3. Hallucination (does it add unsupported claims?)

Score each as pass/fail and explain.
- **Example Result:**
1. Faithfulness: Pass — both claims about hallucination and accuracy are in the context.
2. Completeness: Pass — covers both chunks.
3. Hallucination: Fail — "speeds up inference" is not mentioned in the context and is an unsupported claim.
```

Conclusion

Building a production-ready RAG system requires careful attention to each stage: indexing, retrieval, and generation. The 15 prompts above give you a practical toolkit to test, debug, and optimize your pipeline. Start by using the indexing prompts to improve chunk quality, then move to retrieval prompts to fine-tune search, and finally use generation prompts to ensure faithful and useful answers.

Remember: no single prompt works for every dataset. Experiment with different chunk sizes, overlap percentages, and re-ranking strategies. The best RAG system is one that you continuously refine based on real user queries. Happy building!

References:
- Lewis et al., "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks," NeurIPS 2020.
- Pinecone documentation on hybrid search.
- LangChain guides on chunking and embedding strategies.