Introduction
Retrieval-Augmented Generation (RAG) has become the backbone of modern AI applications, enabling systems to ground large language model (LLM) outputs in real-world data. But even the best vector database or embedding model fails without well-crafted prompts. This article provides 10 concrete, copy-paste-ready prompts for every stage of a RAG pipeline—from chunking and embedding to hybrid search and final answer generation. Each prompt includes a clear task description, a usage example, and practical notes. Whether you are building a customer support bot, a research assistant, or an internal knowledge base, these prompts will help you get the most out of your RAG system.
Why Prompts Matter in RAG
A RAG system consists of three core stages: indexing (preparing and storing data), retrieval (finding relevant documents), and generation (producing an answer). At each stage, the prompt you feed to an LLM determines the quality of the output. For example, a poorly written query for a vector database may return irrelevant chunks; a vague generation prompt can produce hallucinations. By using specialized prompts, you can drastically improve both precision and recall. Below is a table summarizing the stages and the corresponding prompt types.
| Stage | Primary Task | Prompt Focus |
|---|---|---|
| Indexing | Chunking, metadata extraction, embedding | Structural consistency, semantic boundaries |
| Retrieval | Query rewriting, hybrid search (BM25 + vector) | Query clarity, weighting strategy |
| Generation | Answer synthesis, citation, refusal | Grounding, factuality, tone |
Prompts for Indexing
1. Semantic Chunking Prompt
Task: Split a long document into coherent chunks that preserve meaning. This prompt tells the LLM to break text at natural semantic boundaries (e.g., paragraph ends, section headers) rather than fixed character counts.
Prompt:
"""
You are an expert in document preprocessing. Split the following text into chunks. Each chunk must:
- Be between 200 and 500 words.
- End at a natural break (paragraph end, section header, or sentence boundary).
- Not split a table, code block, or list.
- Include the source document title and a short descriptive tag in the format [Title: tag].
Text:
{insert document text}
Return only a JSON array of strings.
"""
Usage Example: Feed a 10-page PDF report. The LLM returns 12 chunks, each with a tag like "Annual Report: Financial Overview." This improves retrieval accuracy because chunks are semantically self-contained.
2. Metadata Extraction Prompt
Task: Extract structured metadata from a chunk (e.g., date, author, keywords, section). Use this to enrich vector store filters.
Prompt:
"""
Extract the following metadata from the chunk below. Return a JSON object with keys: "date" (ISO format if present), "author" (full name), "keywords" (up to 5), "section" (e.g., Introduction, Methodology). If a field is missing, set it to null.
Chunk:
{insert chunk text}
"""
Usage Example: For a chunk "On March 15, 2024, Dr. Smith reported...", the LLM returns {"date": "2024-03-15", "author": "Dr. Smith", "keywords": ["report", "research"], "section": "Findings"}. This allows filtering by date range or author at query time.
3. Embedding Optimization Prompt (for creating synthetic queries)
Task: Generate diverse synthetic queries for a chunk to train or fine-tune an embedding model. This improves vector search by aligning queries with the chunk's content.
Prompt:
"""
Given the following chunk, generate 5 distinct questions that a user might ask to find this exact information. The questions should vary in phrasing and specificity (one broad, one narrow, one comparative, one definitional, one hypothetical).
Chunk:
{insert chunk text}
Return a JSON array of strings.
"""
Usage Example: For a chunk about Python error handling, the LLM generates: ["What are try-except blocks?", "How to handle FileNotFoundError in Python?", "Compare try-except with if-else for error handling", "Define exception propagation", "What would happen if you omit the finally clause?"]. These synthetic queries can be used to train a retrieval model via contrastive learning.
Prompts for Retrieval
4. Query Rewriting Prompt
Task: Rewrite a user's raw query into a more effective search query for a vector database. This handles typos, vague language, and multi-intent queries.
Prompt:
"""
You are a search query optimizer. Rewrite the following user query to maximize retrieval relevance. Rules:
- Correct spelling and grammar.
- Expand abbreviations (e.g., "NLP" → "natural language processing").
- If the query has multiple intents, break it into separate queries separated by semicolons.
- Remove stop words if they don't add meaning.
- Keep the original intent.
User query: "how to fix db conn issues quickly"
Return only the rewritten query string.
"""
Usage Example: Input: "how to fix db conn issues quickly" → Output: "fix database connection issues quickly" or "database connection troubleshooting; fast database connection fix". This reduces retrieval noise and improves top-k relevance.
5. Hybrid Search Weighting Prompt
Task: Determine the optimal weighting between keyword (BM25) and vector search scores for a given query. Useful when you have a hybrid search backend (e.g., Elasticsearch + vector index).
Prompt:
"""
Given the query below, suggest a weighting factor alpha (0 = pure keyword, 1 = pure vector) that best balances exact term matching with semantic similarity. Consider:
- Queries with proper nouns, codes, or numbers → higher BM25 weight.
- Queries with abstract concepts or synonyms → higher vector weight.
- Return a single number between 0 and 1, and a short explanation.
Query: "Error code 0x80070005 in Windows 10"
"""
Usage Example: For the query above, the LLM might return {"alpha": 0.7, "explanation": "Error code is a specific identifier, so BM25 should dominate, but semantic similarity helps for related descriptions."}. You can then use this alpha to combine scores.
6. Query Expansion Prompt for RAG
Task: Expand a query with synonyms and related terms to improve recall, especially for sparse domains.
Prompt:
"""
Expand the following query with 3–5 alternative phrasings or synonyms that would retrieve relevant documents. Keep the core meaning. Return a JSON array of strings.
Query: "machine learning model underfitting"
"""
Usage Example: Output: ["machine learning model underfitting", "high bias in ML model", "model not learning training data", "underfit ML algorithm", "training error too high"]. Expanding the query before embedding can increase recall by 10–20% in niche corpora.
Prompts for Generation
7. Grounded Answer Generation Prompt
Task: Generate an answer strictly based on retrieved chunks, with citations. This is the most critical prompt to prevent hallucination.
Prompt:
"""
You are a factual assistant. Use only the provided context to answer the user's question. If the context does not contain the answer, say "I cannot answer based on the provided information." Include citations in the format [Source:
Context:
{insert retrieved chunks}
User question: {insert question}
Answer:
"""
Usage Example: If the context includes a chunk titled "Annual Report: Revenue 2024" stating revenue grew 15%, the answer should be "Revenue grew 15% in 2024 [Source: Annual Report: Revenue 2024]." This prompt ensures traceability and reduces hallucination.
8. Refusal Prompt for Out-of-Scope Queries
Task: Politely refuse to answer questions outside the document corpus, while offering alternative help.
Prompt:
"""
You are a RAG assistant for a specific knowledge base. If the user's question cannot be answered from the retrieved context, respond with: "I'm sorry, but I don't have information about that in my current documents. You may want to ask about [list 2–3 related topics from the context]." Do not guess.
Retrieved context:
{insert chunks}
User question: {insert question}
Response:
"""
Usage Example: If the user asks "What is the weather today?" and the context only contains financial reports, the assistant responds: "I'm sorry, but I don't have information about the weather. You may want to ask about financial trends, revenue, or market analysis." This maintains trust and avoids off-topic answers.
9. Multi-Step Reasoning Prompt (Chain-of-Thought RAG)
Task: For complex questions requiring multiple retrieval steps, guide the LLM to decompose the question and retrieve iteratively.
Prompt:
"""
You are a multi-step RAG system. For the user's question, follow this plan:
1. Decompose the question into sub-questions.
2. For each sub-question, generate a search query.
3. After receiving retrieved chunks for each sub-query, synthesize the final answer.
4. Provide the final answer with citations.
User question: "How did the company's revenue growth in Q3 compare to its R&D spending over the last two years?"
Decomposition and queries:
"""
Usage Example: The LLM might output: "Sub-questions: (1) What was revenue growth in Q3 2025? (2) What was R&D spending in 2024 and 2025? Queries: 'Q3 2025 revenue growth', 'R&D spending 2024', 'R&D spending 2025'. Answer: Revenue grew 12% in Q3 2025 [Source: Q3 Report], while R&D spending increased from $5M to $6.2M [Source: Budget Reports]." This enables complex analytical questions.
10. Summarization with Key Points Prompt
Task: Generate a concise summary of multiple retrieved chunks, highlighting key facts, numbers, and dates.
Prompt:
"""
Summarize the following chunks in 3–5 bullet points. Each bullet must include a specific fact, a number (if present), and a citation in [Source: title] format. Omit redundant information.
Chunks:
{insert chunk texts}
Summary:
"""
Usage Example: For chunks about a product launch, output: "- The product launched on Jan 15, 2025 [Source: Press Release]. - Initial sales reached 10,000 units in the first month [Source: Sales Report]. - Customer satisfaction score was 4.5/5 [Source: Survey]." This is ideal for dashboards or executive briefs.
Practical Considerations
When implementing these prompts, keep these tips in mind:
- Temperature setting: For indexing and retrieval prompts, use a low temperature (0–0.2) to ensure deterministic output. For generation, you can use 0.3–0.7 for creativity.
- Token limits: Ensure your LLM context window accommodates the chunk plus instructions. For long documents, use a sliding window approach.
- Evaluation: A/B test prompts with a small set of queries. Measure retrieval precision@k and answer correctness. Tools like RAGAS can help automate this.
- Dynamic weighting: In hybrid search, the optimal alpha (from prompt 5) can vary by query. Consider caching the alpha for repeated query patterns.
Conclusion
Crafting effective prompts for RAG systems is not a one-time task—it requires experimentation and tuning for your specific data and use case. The 10 prompts above cover the entire pipeline: from preparing your documents (semantic chunking, metadata extraction, synthetic queries) to retrieving relevant information (query rewriting, hybrid weighting, expansion) and generating grounded answers (citations, refusals, multi-step reasoning, summaries). By applying these prompts, you can significantly improve the accuracy, relevance, and trustworthiness of your RAG application. Start with the generation prompts (7 and 8) to reduce hallucinations, then optimize retrieval with prompts 4–6, and finally enhance indexing with prompts 1–3. The combination will yield a robust, production-ready RAG system.
Comments