What We Learned by Reproducing 2,200 Papers from ICML: A Practical Reproducibility Guide

The machine learning community has a reproducibility problem. Despite decades of research, many published results are difficult — or impossible — to replicate. In 2026, an open community project took on the largest reproducibility effort to date: reproducing 2,200 papers from the International Conference on Machine Learning (ICML). The project’s findings, published in an open report, are both a warning and a roadmap for anyone working in AI research. They also provide a rare, field-scale window into what goes wrong when theoretical results are brought back to earth.

The full report is available online: Source.

What makes this project so important is not simply the sheer number of reproduced papers. It is the emphasis on open process — the team shared not only the final results, but also the intermediate steps, code, and configuration files. This makes the report a valuable learning resource for every machine learning engineer and researcher. This article breaks down the biggest takeaways and turns them into a practical, step-by-step guide you can use to reproduce any research paper — and to make your own work easier to reproduce.

Why Reproducing 2,200 Papers Is a Big Deal

To appreciate the scale, consider one of the more complex result-producing workflows in machine learning: a typical paper can involve thousands of GPU hours, complicated data pipelines, and dozens of hyperparameter choices. Reproducing a single paper is a challenge. Reproducing 2,200 papers required a coordinated community, shared infrastructure, and a well-defined protocol. The project team did not simply run the original code and check the reported numbers. They had to deal with missing artifacts, ambiguous experimental setups, and code written in outdated frameworks. The result is the most comprehensive picture of the reproducibility landscape currently available.

So what are the key lessons? They fall into four categories: hyperparameters, data, environment, and communication.

Lesson 1: Hyperparameters Are the Silent Killer

A paper can present a fantastic model and a solid theoretical contribution, but if the learning rate is not reported precisely, the experiment cannot be replicated. In many papers, only a subset of hyperparameters appears in the main text; the rest are relegated to a supplementary appendix or omitted entirely. The reproduction project reinforces how common this is. Countless papers have been submitted without the exact settings for the optimizer, batch size, warmup steps, or weight decay. In some cases, the original authors have to be contacted for details.

Practical takeaway: Treat hyperparameters as first-class citizens. Report every setting that affects the final result, including the random seed, batch size, optimizer momentum, weight decay, learning rate schedule, and gradient clipping. Even better, include the exact command line used to launch the training run.

Lesson 2: Data Preprocessing Is Under-Documented

Model code is only half the story. The exact split of train, validation, and test sets, the order in which examples are shuffled, the normalization statistics computed from the training set — all of these affect the outcome. The project reinforces that even when a paper provides complete model code, the data pipeline is often described at a high level: “we used standard preprocessing.” That phrase can mean many different things. For instance, image normalization with mean and standard deviation values can be computed from the training set or taken from an external dataset. The two approaches yield different accuracies.

Practical takeaway: When you share code, also share the data preprocessing script. Document every transformation: resizing, cropping, flipping, normalization, augmentation parameters, and the exact order in which they are applied. If you use a public dataset, specify the exact version and how you split it.

Lesson 3: The Environment Matters as Much as the Code

A paper’s code might be perfectly written, but if it relies on an older version of a library, the behavior can change. The same can be said for GPU drivers, hardware types, and even CPU instruction sets. During such a large reproduction effort, the team had to recreate full environments from scratch, sometimes patching old code to work with modern tools. This is one of the most time-consuming and overlooked parts of reproducibility.

Practical takeaway: Always ship a requirements.txt or an environment descriptor. Even better, use a container image that pins not only the Python packages but also the operating system and lower-level libraries. This is the only way to guarantee that your code runs the same way in six months.

Lesson 4: Communication Is Key

The most common failure of reproducibility is not a lack of code — it is a lack of precise, structured documentation. The project stressed that authors often assume too much knowledge on the part of the reader. A good reproducibility package is a complete story: a README that explains the structure, a script that runs all experiments end-to-end, and a log of results that matches the paper. Without these, even a well-written codebase is useless to someone wanting to verify your results.

Practical takeaway: Write documentation as if you are the reader. Have a colleague try to run your code without any instructions other than the README. If they fail, you know what to fix.

A Step-by-Step Guide to Reproducing a Paper

Now, translate those lessons into action. Here is a practical, repeatable process for reproducing any machine learning paper, which you can apply to your own research or to a paper you’re studying.

Step 1: Deconstruct the Paper

Before you touch any code, read the paper carefully and create a checklist of every critical component:

• Model architecture and its exact configuration (e.g., number of layers, hidden size, activation functions).
• Loss function and any auxiliary losses.
• Optimizer and its hyperparameters (learning rate, beta values, epsilon).
• Learning rate schedule (warmup, decay, milestones).
• Regularization techniques (weight decay, dropout, label smoothing, data augmentation).
• Dataset and preprocessing steps.
• Evaluation protocol (metrics, how many runs are averaged).

Create a table like this:

This table will guide your implementation and help you spot gaps.

Step 2: Locate the Original Code

Search for a repository linked in the paper. If nothing exists, check for community reimplementations. For popular papers, you can often find multiple implementations in different frameworks. When you find code, don’t assume it is up-to-date. Look for commits, issues, and pull requests that might indicate known bugs.

If the original code is unavailable, you have to implement the model from the description. This is slower, but it teaches you the most and helps you build a deeper intuition.

Step 3: Set Up an Isolated Environment

Use a fresh virtual environment or a container. This prevents conflicts with your existing projects. Here’s a minimal shell script to get started:

# Create a project directory
mkdir paper-reproduction
cd paper-reproduction

# Create a virtual environment
python -m venv venv
source venv/bin/activate

# Upgrade pip
pip install --upgrade pip

# Install the dependencies you identified
# (replace with the actual requirements from the project)
pip install <framework> <dataset-tools> <metrics>

# Set a fixed random seed for reproducibility
export PYTHONHASHSEED=42

If you are using a container, write a container definition file that installs the exact versions from the paper. This is more robust, especially when the paper uses an outdated version of a library.

Step 4: Run the Original Code and Verify the Baseline

If you have the original code, run it as-is with the exact configuration from the paper. Do not try to “improve” it. The goal is to see if you can match the reported numbers.

Log everything: the command you used, the environment, and the output. Use a fixed random seed. If the code does not have a seed, add one. If the paper gives a confidence interval, note that.

If you cannot reach the reported accuracy, there could be many reasons:

• The precision of your floating-point calculations differs.
• The GPU type affects non-deterministic operations.
• The data was preprocessed differently.

Once you have your baseline, move to the next step.

Step 5: Build Your Own Implementation

If you are re-implementing the model, start from the model definition. Write the forward pass exactly as described, then the training loop, then the evaluation. Keep track of every decision.

A good practice is to write a simple unit test for each component. For example, if the model uses a residual connection, verify that the output shape matches the expected shape.

Here’s a generic training loop that follows the paper’s description (you’ll need to adapt to your specific paper):

# Generic training loop (adapt to your framework)
model = build_model()
optimizer = create_optimizer(
    model.parameters(),
    lr=0.001,
    betas=(0.9, 0.999)
)
criterion = get_loss_function()

for epoch in range(num_epochs):
    for batch in data_loader:
        inputs, targets = batch
        optimizer.zero_grad()
        outputs = model(inputs)
        loss = criterion(outputs, targets)
        loss.backward()
        optimizer.step()

This is a generic snippet; the important thing is to mirror the paper’s exact optimization settings.

Step 6: Compare and Iterate

Run your implementation and compare the results to the baseline. If they differ, isolate the cause. Is it in data preprocessing? Model initialization? The learning rate schedule? Change one variable at a time and re-run.

Keep a log of your experiments. A simple CSV file can be very useful:

This table is your reproducibility log.

Step 7: Share Your Results

The final step is to make your reproduction public. Write a report that includes:

• A link to the original paper and code.
• Your environment and setup instructions.
• The commands you ran.
• A comparison of your results to the paper
  and a discussion of any discrepancies you observed, along with possible explanations. Include your code repository link so others can inspect and run your work. The goal is not to prove the paper wrong, but to document what you did, what you found, and what you learned.

Once your report is ready, consider sharing it in a public forum: a blog post, a GitHub repository with a README, or a short paper. You can also submit it to a reproducibility track if the conference offers one. Even if you choose not to publish formally, write the report for yourself — it will help you recall the details months later and give you a reference for future projects.

Final Thoughts

Reproducing a research paper is one of the best ways to dive deep into a new field. It forces you to read the details, make concrete decisions, and debug your understanding of the model. The process is rarely smooth: you will encounter missing hyperparameters, ambiguous descriptions, and code that behaves differently than expected. That is normal. The value lies in the journey — every mismatch gives you insight into how the original authors likely made their choices.

Remember to be patient. Start with a small version of the model, test each component, and gradually scale up. Use the reproducibility log to track your experiments and avoid repeating the same mistake twice. And finally, always give the original authors credit — their work and code are the foundation of your own effort.

Now go pick a paper, set up your environment, and start reproducing. It will be frustrating, educational, and ultimately rewarding.

Happy experimenting!P.S. If you're looking for a starting point, I've compiled a short list of papers that are particularly well-suited for a first reproduction attempt: they have clean baselines, manageable datasets, and at least one official or community implementation you can check yourself against. You'll find the list—along with a template for the reproducibility log and a collection of useful debugging scripts—in the resources folder of my repository. Feel free to fork it, adapt it, and make it your own.

And if you do write up your reproduction, I'd genuinely love to read it. Share the link in the comments below, or tag me on social media. There's something special about seeing a paper through someone else's eyes—the parts they struggled with, the shortcuts they found, and the small