Building a Retrieval-Augmented Generation (RAG) system is only half the battle. The other half is proving it actually works. Many teams fall into the trap of reporting impressive accuracy numbers that evaporate in production. The root cause is often an evaluation set that is too small, too easy, or that contradicts how real users interact with the system. A recent article on Habr Source explores this problem and offers practical guidance on assembling your first 100–300 honest RAG evaluation cases. This review distills the key principles, augmented with common practices from the RAG evaluation community.
Why RAG evaluation is deceptive
Standard RAG pipelines are typically measured with metrics like faithfulness, answer recall, and citation precision. However, these numbers only mean something if the evaluation set is representative. If you fill your set with simple questions that have exact matches in the source documents, the model will look great on paper and fail on messy real-world queries. A common failure is "eval set overfitting": repeatedly tuning the system against a small set of golden cases until the metrics look perfect, while generalization collapses.
The Habr article emphasizes that the quality of the eval set outweighs its size. A curated collection of hundreds of cases that reflect real usage patterns is far more valuable than thousands of artificially generated trivia.
Why 100–300 cases?
The recommended range of 100–300 cases is a pragmatic balance. With fewer cases, any metric you compute has very wide confidence intervals, so a 5% difference between two runs is statistically meaningless. With more cases, the cost of manual curation and quality control becomes prohibitive, and annotator fatigue degrades the gold labels. Within this range, you can still manually review each case and maintain a high-quality dataset.
Here is what enough cases enable you to do:
- Cover the most common question types as well as several rare edge cases.
- Include a meaningful number of negative cases (questions that the system should abstain from answering).
- Split the data into development and held-out test sets without losing statistical power.
How to build the eval set
The Habr article suggests a practical workflow. Start with real user queries from your application logs. These are the ground truth of what your system will face in production. Tag each query with its expected failure mode: simple lookup, multi-hop reasoning, temporal, ambiguous, or out-of-scope.
Then, synthesize additional cases to fill known gaps. For example, if your knowledge base contains product documentation, generate questions that require combining information from several sections. If you are using OpenAI's API to generate paraphrases, you can automate this step, but every synthetic case must be human-validated, because LLM-generated questions often carry subtle biases.
A crucial and often overlooked category is the negative case — a question that is outside the knowledge base, such as "How do I repair my car engine?" when the system is a software support bot. A robust RAG system should explicitly refuse to answer rather than hallucinate. Make sure your eval set includes a substantial share of these scenarios.
| Category | Example | Why it matters |
|---|---|---|
| Simple factual | "What is the refund policy?" | Baseline sanity check |
| Multi-hop reasoning | "Compare the battery life of Model A and Model B." | Tests retrieval synthesis |
| Negative | "How do I replace a car engine?" | Tests abstention and root detection |
Avoiding self-deception
Even a well-built eval set can be misused. The article articulates three ground rules:
- Never tune and test on the same cases. Hold out a fixed test set, and touch it only for final evaluation. Tuning against the test set, even implicitly, pollutes the measurement.
- Measure failure modes separately. Overall accuracy can be 85% while multi-hop questions have only a 50% success rate. Always break down metrics by query category.
- Version your eval set. When you add new documents or change the system prompt, the eval set must be updated. But do it through a formal versioning process and never alter gold answers to match current model outputs — that is "gold answer drift."
Another practical recommendation is to start with a pilot set of 30–40 manually curated queries, then expand iteratively. This lets the team align on annotation guidelines before scaling to hundreds of cases.
Common mistakes in eval set construction
Even experienced developers fall into these traps:
- Curation bias: picking only questions that are interesting to the team, not to the users.
- Ignoring abstention: omitting out-of-scope questions, which makes the system look more confident than it really is.
- Gold label leakage: reading model outputs before writing gold answers, then unconsciously aligning them.
- Rigid set: never updating the eval set when the knowledge base grows, so the system overfits to obsolete data.
- Hiding the variance: reporting only the average metric while ignoring the distribution across categories.
Practical next steps
To operationalize this, you can use frameworks like Ragas or LangSmith to log and evaluate experiments. But the eval set itself must be domain-specific and honest. The investment in building a good set pays off by making every future improvement measurable and meaningful.
If you are building an automated pipeline that relies on LLMs to generate synthetic cases, consider a platform that orchestrates these calls. ASI Biont supports connecting to OpenAI via API — more details at asibiont.com/courses.
The bottom line
An honest RAG eval set is not a luxury; it is the only way to trust your system in production. Start with a modest number of high-quality cases, prioritize real user queries, and treat the set as a living artifact that evolves alongside your system. The goal is not to chase a higher number, but to have a number you can believe.
Comments