How We Measured AI Writing Across arXiv, and Where the Measurement Breaks

Introduction

In July 2026, a team of independent researchers published a detailed analysis titled “How we measured AI writing across arXiv, and where the measurement breaks,” on the blog Unslop.run. The study set out to quantify the prevalence of AI-generated text in academic preprints on arXiv, a repository that hosts over 2.5 million papers in physics, mathematics, computer science, and related fields. The researchers aimed to provide a data-driven answer to a question that has haunted the scientific community since the release of large language models (LLMs) like GPT-3.5 and GPT-4: How much of the new scientific literature is actually written by machines?

The investigation used a custom detection pipeline that combined statistical classifiers, perplexity scoring, and a novel “burstiness” metric to flag unusually uniform prose. The results were striking: depending on the subdomain, between 8% and 22% of recently submitted abstracts showed strong signatures of AI assistance. However, the authors were equally candid about the method’s blind spots, revealing where the measurement breaks down—and why perfect detection may remain elusive.

This article unpacks the methodology, presents key findings, and explores the limitations that every practitioner should understand before relying on AI detection in academic writing.

The Measurement Pipeline: What the Researchers Built

The team developed a three-stage detection pipeline, each stage targeting a different hallmark of LLM-generated text.

Stage 1: Classifier-Based Filtering

First, they trained a lightweight transformer classifier on a balanced dataset of human-written and AI-generated abstracts (sourced from arXiv pre-2022 for human text, and from GPT-4 outputs prompted with the same titles). The classifier achieved an F1-score of 0.89 on a held-out test set. It was then applied to the entire corpus of arXiv abstracts submitted between January 2023 and June 2026.

Stage 2: Perplexity Scoring

For each abstract flagged by the classifier as “likely AI,” the team computed the average per-token perplexity using a reference language model (a fine-tuned version of Llama 3 70B). The intuition is that LLMs tend to generate text with lower perplexity (higher predictability) than human writing. A threshold was set at perplexity < 8.5 for a 100-token window.

Stage 3: Burstiness Analysis

The most innovative step involved measuring “burstiness”—the variance in word frequency distribution. Human writing exhibits greater burstiness: humans repeat certain words in clusters (e.g., “we propose” three times in one paragraph) while LLMs distribute repetitions more evenly. The team used the Gini coefficient of word occurrence intervals to quantify this. Abstracts with a Gini coefficient below 0.35 were flagged as machine-like.

Only abstracts that passed all three stages were counted as “definitively AI-assisted.” The authors estimate that this conservative approach underestimates the true prevalence by a factor of 1.5 to 2.

Key Findings: The Numbers That Surprised Everyone

The analysis covered 1.2 million abstracts submitted from January 2023 to June 2026. The overall proportion flagged as AI-assisted was 12.4%. But the distribution was far from uniform.

• Computer Science (CS): The highest rate at 22.1%, especially in subfields like natural language processing (NLP) and machine learning (ML).
• Electrical Engineering and Systems Science: 18.7%.
• Physics and Mathematics: 8.3% and 7.9%, respectively.
• Quantitative Biology: 10.5%.

The temporal trend was also revealing. In early 2023, fewer than 3% of abstracts were flagged. By mid-2026, the monthly rate had climbed to over 18% across all categories. The researchers attributed this acceleration to the widespread adoption of ChatGPT, Claude, and open-source models like Llama 3 and Mistral.

One particularly striking data point: among papers whose authors were affiliated with institutions in the top 20 of the Shanghai Ranking, the AI-assisted rate was 14.9%, versus 11.2% for all other institutions. This suggests that AI writing tools are being adopted even by elite researchers, not just those under pressure to publish quickly.

Where the Measurement Breaks: Five Critical Limitations

The researchers devoted a substantial section of their report to the method’s failures. This transparency is rare and valuable.

1. Human-Written Abstracts That Mimic AI

Some human authors deliberately adopt a flat, repetitive style—often due to strict journal formatting guidelines or non-native English writing. Such abstracts produce low perplexity and low burstiness scores, leading to false positives. The team estimated that 2–3% of human-written abstracts are misclassified as AI-generated.

2. Hybrid Writing (Human + AI Polish)

The pipeline assumes a binary classification: fully human or fully AI. In reality, many researchers use AI to rephrase individual sentences or improve grammar. These hybrid texts often retain human-level burstiness in content but show AI-like perplexity in style. The three-stage pipeline cannot reliably separate them.

3. Evolving Language Models

Models like GPT-4o and Claude 3.5 Sonnet generate text with higher perplexity and greater burstiness than earlier GPT-3.5 outputs. As LLMs improve, they become harder to distinguish from human writing. The authors noted that their classifier’s performance degraded by 15% when tested on abstracts generated by models released after June 2025.

4. Short Text Challenges

arXiv abstracts are typically 150–300 words. Statistical measures like perplexity and Gini coefficient require a minimum token count to be stable. The team found that for abstracts shorter than 100 tokens, the false positive rate increased to 12%. This is a fundamental limitation: detection methods that work on full articles (500+ words) cannot be directly applied to short snippets.

5. Adversarial Evasion

The researchers acknowledge that any public detection method can be reverse-engineered. A user could prompt an LLM with instructions to “increase burstiness” or “add rare words,” easily evading the fixed thresholds. The team did not attempt to build an adversarial detector, but they note that the arms race between generators and detectors is already underway.

Practical Implications for the Scientific Community

The study’s findings have direct consequences for peer review, plagiarism detection, and the integrity of scientific publishing.

• Reviewers should not rely on any single detection tool. The researchers recommend a multi-stage approach similar to theirs, but also advise manual inspection of papers flagged by only one or two stages.
• Journal editors may need to update submission guidelines to require disclosure of AI assistance. As of July 2026, fewer than 30% of top-tier journals have explicit AI writing policies.
• Conference organizers in CS and engineering fields should expect that a significant fraction of submissions involve AI writing. The authors suggest that conferences like NeurIPS and ICML commission dedicated studies to measure the impact on review quality.

For developers and data scientists building their own detection systems, the article offers a reproducible pipeline on GitHub (linked in the source). The code includes the classifier weights, perplexity calculation scripts, and burstiness analysis functions. The team also provides a public dashboard where users can test individual abstracts against their model.

Conclusion

“How we measured AI writing across arXiv, and where the measurement breaks” is a landmark contribution to the ongoing debate about AI’s role in academic writing. By combining classifier, perplexity, and burstiness metrics, the researchers produced the most comprehensive estimate to date: approximately 12% of recent arXiv abstracts are likely AI-assisted, with peaks above 20% in computer science. Yet the study is equally valuable for its honesty about blind spots—short texts, hybrid authorship, evolving models, and adversarial evasion all limit the reliability of detection.

For the scientific community, the takeaway is clear: AI writing is now a permanent feature of the scholarly landscape. Rather than pursuing perfect detection, stakeholders should focus on transparency, disclosure norms, and updated quality assurance processes. The measurement may break, but the conversation it sparks is essential.

Source