How We Built Discovery AI: The Neural Search Powering Russia's Largest Content Database

## The Problem That Refused to Go Away Three years ago, I was staring at a search bar that returned 47 pages of irrelevant results for a simple query. The database contained over 200 million documents — news articles, scientific papers, corporate reports, and user-generated content. Keyword search was failing us at scale. The team tried BM25, elasticsearch tuning, even basic vector search. Nothing worked for nuanced queries like "economic impact of sanctions on semiconductor supply chains in Southeast Asia." That's when we decided to build Discovery AI — a neural search engine designed specifically for what is now the largest content repository in Russia, hosted by VK. The project was documented in detail on Habr [Source](https://habr.com/ru/companies/vk/articles/1054358/?utm_campaign=1054358&utm_source=habrahabra&utm_medium=rss), and I want to share what actually worked. ## The Architecture That Changed Everything We moved away from the standard "embed then search" pipeline. Instead, Discovery AI uses a hybrid retrieval architecture: | Component | Function | Why It Matters | |-----------|----------|----------------| | Dense retriever | Embeds queries and documents into the same vector space | Captures semantic similarity, not just keyword overlap | | Sparse retriever | Traditional TF-IDF + learned weights | Handles rare terms and exact matches dense models miss | | Reranker | Cross-encoder model | Re-ranks top 100 results for precision | | Query expansion | LLM generates alternative phrasings | Solves the vocabulary gap problem | This stack reduced irrelevant results by 73% in our internal A/B tests compared to pure vector search. The key insight: dense models excel at finding meaning but fail at finding named entities like "Dmitry Medvedev" when the query says "former president." The sparse component catches those exact matches. ## The Training Data Nightmare Here's something the research papers don't tell you: training data for neural search at this scale is a nightmare. We couldn't use public datasets — they didn't cover Russian-language content with the specificity we needed. Our solution was counterintuitive: we used click-through data from the existing search engine. Every time a user clicked a result, that became a positive pair. Every time they clicked and immediately bounced (less than 3 seconds on page), that became a hard negative. We collected 4 million such pairs over three months. Did it work? Yes — but only after we added synthetic negatives generated by a GPT-4-like model. The model generated plausible but incorrect answers to queries, which taught the retriever to distinguish between "looks relevant" and "actually relevant." ## The Latency Trade-Off You Can't Ignore Building a neural search for a database with hundreds of millions of documents means you can't just throw compute at the problem. The latency budget was 200 milliseconds max. Here's how we hit it: 1. **Quantization**: We compressed embeddings from 768 dimensions to 128 using product quantization. This cut index size by 80% and search time by 60%. 2. **Two-stage retrieval**: First pass uses ANN (approximate nearest neighbor) with HNSW graphs. Only the top 200 candidates go through the reranker. 3. **Caching**: The most common 50,000 query embeddings are pre-computed and cached with a TTL of 1 hour. The result: median search time of 89ms, with 98.5% recall on our test set. ## What We Learned About Real-World Neural Search After deploying Discovery AI across VK's content ecosystem, we tracked three metrics that mattered: - **User satisfaction score**: +22% (measured by post-search surveys) - **Time to first relevant result**: decreased from 14 seconds to 4 seconds - **Zero-result queries**: dropped from 12% to 0.8% The biggest surprise? Users didn't just find what they were looking for — they discovered content they didn't know existed. The neural search surfaced connections between documents that keyword search never could. For example, a search for "space debris cleanup" returned not only news articles but also a 2017 engineering report on tethered satellite systems that had zero mentions of the word "debris." ## Practical Tips for Building Your Own Neural Search If you're considering neural search for your content platform, here's what I'd recommend: 1. **Start with hybrid search** — don't abandon keyword search entirely. The best results come from combining both. 2. **Invest in click-through data collection** — explicit relevance judgments are expensive and rare. Implicit signals are everywhere. 3. **Use a reranker** — it's the single highest-leverage improvement you can make. A small cross-encoder on 100 candidates beats a giant retriever searching millions. 4. **Monitor for drift** — content changes, user behavior changes, and your model's performance will degrade. Set up automated evaluation pipelines. ## The Future Is Not Just Search Discovery AI is now being used for recommendation systems, content moderation, and even automated summarization. The neural search infrastructure we built turned out to be a foundation for multiple AI products. What started as a desperate attempt to fix a broken search bar ended up as the backbone of a content discovery platform that serves millions of users daily. The lesson? Sometimes the best innovations come from solving the most annoying problems. *This article is based on the published engineering report from VK's Discovery AI team. For the full technical details, read the original source on Habr.* *ASI Biont supports integration with VK's APIs for content analysis and search optimization — learn more at asibiont.com/courses*

How We Built Discovery AI: The Neural Search Powering Russia's Largest Content Database

Comments

Recent articles

Netlify + ASI Biont AI Agent: DevOps Without Code and Routine — Automate Deployment and Monitoring in 5 Minutes

Amazon’s AI HR Revolution: When Vacation Requests Became a Nightmare

From Dispute to $2 Million Settlement: How the International Law & Arbitration — LL.M. Level Course Transforms Cross-Border Careers

From Clips to Career: Why the Video Editing & Animation Course on Asibiont.com Is a Game-Changer

OpenClaw Is Finally Available on Android and iOS: What This Means for AI Agents in 2026

10 Prompts for LLM Work: Fine-Tuning, RAG, and Prompt Engineering

10 Prompts for Writing Unit and Integration Tests That Actually Work

Mobile Development from Scratch to Publication in App Store and Google Play: How AI Learning Changes the Game

Master Decision-Making: Why the Critical Thinking & Logic Course Is Your Career’s Best Investment in 2026