Introduction
In a detailed analysis published on Habr, a developer examined 591 responses from ChatGPT's search feature to understand patterns in how the AI cites sources. The study found that one-third of all user queries to ChatGPT's search function are repetitive—meaning users ask the same or very similar questions multiple times. More importantly, the analysis reveals a stark disparity: a small group of sources is cited in almost every relevant response, while the vast majority of sources appear only sporadically. This article breaks down the methodology, key findings, and implications of this research, offering insights for content creators, SEO specialists, and AI users.
The Study: Methodology and Scope
The researcher collected 591 responses generated by ChatGPT when it used its built-in search capability (the feature that allows the model to fetch live data from the web). Each response was parsed to extract the list of cited URLs. The goal was to determine:
- How many distinct sources are cited across all responses?
- How often does each source appear?
- What fraction of queries are semantically similar or identical?
Key Metrics from the Dataset
| Metric | Value |
|---|---|
| Total responses analyzed | 591 |
| Unique queries (after deduplication) | ~394 |
| Fraction of repetitive queries | ~33% (197 out of 591) |
| Total unique domains cited | 142 |
| Domains cited in >5% of responses | 12 |
| Top 3 domains by citation frequency | Wikipedia, BBC, Reuters |
Note: The study classified queries as "repetitive" if the core intent and key terms were identical, even if wording differed slightly.
Why One-Third of Queries Are Repetitive
The high proportion of repeated queries can be attributed to several factors:
1. User behavior: Many people ask the same question with minor variations (e.g., "What is the capital of France?" vs. "Capital of France 2025").
2. Session continuity: Users often refine a query within a single conversation, leading to near-identical searches.
3. Template-based queries: Common use cases like "latest news on [topic]" or "definition of [term]" generate repetitive patterns.
The Citation Inequality: A Few Sources Dominate
Perhaps the most striking finding is the unequal distribution of citations. The analysis reveals a classic Pareto-like pattern: roughly 20% of cited domains account for 80% of all citations. Specifically:
- Wikipedia appears in 47% of all responses that require factual or encyclopedic information.
- BBC News is cited in 34% of news-related queries.
- Reuters follows with 29% of news responses.
- The remaining 130+ domains are cited in fewer than 5% of responses each, with many appearing only once.
Why This Happens
The researcher suggests several reasons for this concentration:
- Domain authority: ChatGPT's retrieval system appears to prioritize well-established, high-authority domains. Wikipedia's consistent structure and broad coverage make it a default choice.
- Training data bias: The model's underlying training data (which includes web crawls) may have overrepresented certain sources, influencing the search retriever's ranking.
- Speed vs. diversity: To generate responses quickly, the system may rely on a shortlist of trusted sources rather than searching broadly each time.
Implications for Content Creators and SEO
For those who produce content intended to be cited by AI models, the data offers both sobering news and actionable guidance:
- Getting into the "always cited" club is extremely hard. The top sources are already entrenched and rarely replaced.
- Niche content has a chance if it answers a specific, high-value query that general sources don't cover. For example, a specialized technical blog might be cited for a unique API documentation question.
- Structured data and clarity help. The analysis noted that pages with clear headings, bullet points, and concise answers were more likely to be extracted as citations.
Practical Recommendations
- Target long-tail queries that are not already dominated by Wikipedia or major news outlets.
- Optimize for direct answers: Use FAQ sections, tables, and summary boxes.
- Build domain authority through consistent, high-quality content and backlinks from reputable sites.
Technical Details: How ChatGPT Selects Sources
While the exact algorithm is proprietary, the study's author reverse-engineered some behaviors:
- The retriever uses a dense passage retrieval model (likely based on a fine-tuned transformer) to find relevant snippets.
- Recency is not a strong factor—older but authoritative articles are still preferred over breaking news from obscure blogs.
- Language consistency matters: The model tends to cite sources in the same language as the query, even if a better source exists in another language.
Conclusion
The Habr analysis provides a rare, data-driven glimpse into how ChatGPT's search citation system works—and doesn't work. The finding that a third of queries are repetitive suggests that users are either not getting the answer they need on the first try, or they are habitually asking the same questions. The extreme concentration of citations among a few sources raises questions about diversity of information and potential for echo chambers. For content creators, the path to being cited is narrow but not impossible: focus on underserved niches, structure content for easy extraction, and build genuine authority over time.
Comments