In the rapidly evolving landscape of AI-driven education, a curious paradox has emerged: while large language models (LLMs) like GPT-4 and Claude dominate headlines, some of the most effective adaptive learning systems deliberately avoid them. A recent technical deep-dive published on Habr reveals exactly such a system—an interview preparation trainer that relies on a hybrid of the Leitner system, Elo ratings, and guess correction algorithms to schedule questions and assess user knowledge. This article examines the architecture, trade-offs, and empirical results of this approach, drawing directly from the project team's documented experience.
The Problem: Why Not Use an LLM for Everything?
The developers behind this project faced a common challenge in edtech: how to build a scalable, low-latency interview trainer that could personalize question difficulty without expensive GPU compute or unpredictable API costs. Many modern trainers offload reasoning to LLMs, but this introduces several issues:
- Latency: Real-time question adaptation requires sub-second responses, which LLM inference cannot guarantee.
- Cost: Each LLM call adds marginal expense, making free-tier or high-volume usage prohibitive.
- Determinism: LLMs produce non-deterministic outputs, complicating fair assessment in a testing context.
- Bias and Hallucination: Models may generate incorrect or biased explanations, undermining trust.
The article's authors describe a conscious decision to build a deterministic engine that never calls an LLM during user sessions. Instead, they combine three well-established algorithms—Leitner, Elo, and a statistical guess correction—to achieve adaptive difficulty and accurate knowledge estimation.
The Solution: A Three-Engine Architecture
The resulting system is not a single algorithm but a pipeline of three components, each addressing a different subproblem:
| Component | Role | Key Parameter |
|---|---|---|
| Leitner System | Schedules question repetition based on correctness and time | Box intervals (1, 2, 4, 8… days) |
| Elo Rating | Adjusts question difficulty relative to user skill | K-factor = 32, initial rating = 1500 |
| Guess Correction | Penalizes lucky answers and rewards consistent performance | Correction factor = 0.25 |
Leitner System: Spaced Repetition for Interview Topics
The Leitner system, originally developed by Sebastian Leitner in the 1970s for flashcard learning, divides questions into boxes based on how often they should be reviewed. In this implementation, the team uses five boxes:
- Box 1: Questions answered incorrectly once—reviewed daily.
- Box 2: Correct once—reviewed every 2 days.
- Box 3: Correct twice consecutively—reviewed every 4 days.
- Box 4: Correct three times—reviewed every 8 days.
- Box 5: Mastered—reviewed every 16 days (or on demand).
A critical innovation is the integration of Elo ratings to decide which questions from a box to show first, rather than random selection. The article notes that without this, users would see trivial or overly hard questions in the same box, reducing learning efficiency.
Elo Rating: Dynamic Difficulty Adjustment
Elo, originally designed for chess rankings, is repurposed here to model the interaction between a user's skill and a question's difficulty. The update formula is:
Expected_score = 1 / (1 + 10^((Question_rating - User_rating) / 400))
New_user_rating = Old_user_rating + K * (Actual_score - Expected_score)
New_question_rating = Old_question_rating + K * (Expected_score - Actual_score)
Where K = 32 (standard for rapid convergence) and Actual_score is 1 if correct, 0 if incorrect. The team seeded all questions with a starting Elo of 1500, same as new users. After each answer, both ratings update, creating a zero-sum game that pushes questions toward the user's current level.
Real-world example: A user answers a question initially rated 1500 correctly. The expected score for a 1500-rated user against a 1500-rated question is 0.5. The actual score is 1.0, so the user gains 32 * (1.0 - 0.5) = 16 points, reaching 1516. The question loses 32 * (0.5 - 1.0) = -16 points, dropping to 1484. Next time, the system will present a slightly harder question (rating ~1516) from the same Leitner box.
Guess Correction: Eliminating Lucky Answers
A persistent problem in multiple-choice assessments is guessing. A user who answers correctly by chance receives an undeserved rating boost. The article describes a guess correction factor applied to the score before the Elo update:
Corrected_score = (Actual_score - Guess_probability) / (1 - Guess_probability)
For a 4-option multiple-choice question, Guess_probability = 0.25. If the user answers correctly, Corrected_score = (1 - 0.25) / (1 - 0.25) = 1.0 (no penalty). But if they answer incorrectly, Corrected_score = (0 - 0.25) / (1 - 0.25) = -0.333. This penalizes wrong answers more heavily, making it harder to climb via random clicks.
For open-ended questions (no guessing), Guess_probability = 0, so Corrected_score = Actual_score.
Results: What the Data Shows
The developers ran a controlled experiment with 500 simulated users, each answering 200 questions. Key metrics:
- Convergence time: The system stabilized user ratings within 50 questions (vs. 80+ without Elo).
- Question coverage: 95% of questions in the database were used (vs. 40% with random selection from Leitner boxes).
- Guess rate: Detected 22% of correct answers as probable guesses (corrected before rating update).
| Metric | Before (Random + Leitner) | After (Leitner + Elo + Correction) |
|---|---|---|
| Questions to converge | 80 | 50 |
| Database utilization | 40% | 95% |
| Guess detection rate | 0% | 22% |
| User rating variance | ±200 points | ±50 points |
Practical Implications for Interview Training
This architecture offers several advantages for real-world interview preparation:
1. Zero inference cost: Entirely deterministic, runs on a single CPU core for thousands of concurrent users.
2. Fair assessment: No LLM bias; all users receive identical algorithmic treatment.
3. Scalability: Adding questions or users requires no retraining—just database updates.
4. Transparency: Every rating change can be logged and audited, unlike black-box LLM outputs.
The project team reports that the system has been running in production for six months, processing over 10 million answers with 99.9% uptime and no LLM calls.
Conclusion
The article demonstrates that a well-designed combination of classic algorithms—Leitner for repetition, Elo for difficulty, and guess correction for accuracy—can create an adaptive interview trainer that rivals LLM-based systems in personalization while outperforming them in cost, speed, and determinism. For developers building educational tools, this approach offers a proven alternative to the hype around generative AI. The full source code and dataset are linked in the original article.
Comments