How to Automate Model Retraining and Deployment with GitHub Actions and MLflow: Step-by-Step Guide

How to Automate Model Retraining and Deployment with GitHub Actions and MLflow: Step-by-Step Guide

Introduction

Two years ago, a mid-sized fintech company deployed a fraud detection model. It worked flawlessly for three months—then false positives skyrocketed. The team spent two weeks manually retraining, testing, and redeploying. By then, the damage was done: angry users, lost transactions, a bruised reputation.

This story repeats across industries. The gap between model development and production reality grows with every data drift, every new feature, every business rule change. The solution isn't more manual oversight—it's automation. Specifically, a CI/CD pipeline for machine learning that retrains models on fresh data and redeploys them without human intervention.

In this guide, you'll learn how to build exactly that using GitHub Actions (free tier) and MLflow (open source). By the end, you'll have a production-ready pipeline that automatically retrains your model on a schedule or trigger, logs all experiments, and deploys the best version to a staging environment. No expensive tools required.

If you want to go deeper—covering feature stores, A/B testing, Kubeflow, data drift monitoring, and cost optimization—check out the full Production ML (MLOps) course on asibiont.com, which covers everything from model serving to hyperparameter tuning in production settings.

Why Automate Model Retraining and Deployment?

Manual retraining and deployment create several pain points:

Automation with CI/CD for machine learning solves all of these. A GitHub Actions ML pipeline can:

• Retrain on a cron schedule (daily, weekly) or on demand (e.g., when new data arrives)
• Log every run with MLflow for full traceability
• Automatically deploy the best model to a staging server
• Roll back to a previous version if validation fails

This is MLOps automation at its core: reducing manual overhead while ensuring production models stay current.

Prerequisites

Before we dive into code, make sure you have:

• A GitHub repository with your ML code (Python, scikit-learn or TensorFlow)
• MLflow Tracking Server (can be local or on a cloud VM)
• A staging server (e.g., a small AWS EC2 instance or a Docker host)
• Basic familiarity with YAML and Python

If you're new to MLflow, think of it as an experiment tracker + model registry that also handles deployment. GitHub Actions is your orchestrator.

Step 1: Set Up MLflow Tracking and Model Registry

First, configure MLflow to log experiments and store models. In your training script (train.py), add:

import mlflow
import mlflow.sklearn
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import accuracy_score

mlflow.set_tracking_uri("http://your-mlflow-server:5000")
mlflow.set_experiment("fraud-detection")

with mlflow.start_run():
    model = RandomForestClassifier(n_estimators=100)
    model.fit(X_train, y_train)
    predictions = model.predict(X_test)
    acc = accuracy_score(y_test, predictions)

    mlflow.log_param("n_estimators", 100)
    mlflow.log_metric("accuracy", acc)
    mlflow.sklearn.log_model(model, "model")

    # Register the model (creates a new version)
    mlflow.register_model(
        f"runs:/{mlflow.active_run().info.run_id}/model",
        "fraud-detection-model"
    )

This registers every trained model in MLflow's Model Registry. Each version gets a unique ID, metrics, and parameters.

Step 2: Create a GitHub Actions Workflow for Retraining

Now, let's automate retraining. Create .github/workflows/retrain.yml in your repo:

name: Retrain and Deploy ML Model

on:
  schedule:
    - cron: '0 6 * * 1'  # Every Monday at 6 AM UTC
  workflow_dispatch:  # Allow manual trigger

jobs:
  retrain:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: '3.11'

      - name: Install dependencies
        run: |
          pip install -r requirements.txt
          pip install mlflow

      - name: Retrain model
        env:
          MLFLOW_TRACKING_URI: ${{ secrets.MLFLOW_TRACKING_URI }}
        run: |
          python train.py

      - name: Get best model version
        id: best_model
        run: |
          BEST_VERSION=$(mlflow models list --model "fraud-detection-model" --latest 1 --format json | jq -r '.[0].version')
          echo "best_version=$BEST_VERSION" >> $GITHUB_OUTPUT

      - name: Deploy to staging
        env:
          DEPLOY_HOST: ${{ secrets.DEPLOY_HOST }}
          DEPLOY_KEY: ${{ secrets.DEPLOY_KEY }}
        run: |
          # Example: deploy via SSH to staging server
          mlflow deployments create --flavor python_function \
            --model-uri "models:/fraud-detection-model/${{ steps.best_model.outputs.best_version }}" \
            --target $DEPLOY_HOST

Key points:

• The workflow runs on a cron schedule (schedule) and can be triggered manually (workflow_dispatch)
• Secrets like MLFLOW_TRACKING_URI and DEPLOY_HOST are stored in GitHub Secrets
• The mlflow deployments create command deploys the latest registered model to a remote server—this is the model retraining automation part

Step 3: Validate Before Deployment

Deploying a bad model is worse than deploying nothing. Add a validation step before deployment:

- name: Validate model
  run: |
    python validate.py  # Script that runs on a holdout set
    if [ $? -ne 0 ]; then
      echo "Validation failed. Skipping deployment."
      exit 1
    fi

Your validate.py might check:

• Accuracy above a threshold (e.g., 0.85)
• No data drift detected (e.g., using Evidently or Great Expectations)
• Inference time under 100ms

If validation fails, the pipeline stops—no deployment. This is a core MLOps automation best practice.

Step 4: Automate Data Freshness Checks

Model retraining is useless if you retrain on stale data. Integrate a data freshness check into your pipeline. For example, use a Python script that checks the timestamp of your latest training data:

import pandas as pd
from datetime import datetime, timedelta

data = pd.read_parquet("s3://your-bucket/training_data.parquet")
latest_timestamp = data["event_date"].max()

if latest_timestamp < datetime.now() - timedelta(days=7):
    print("Data is stale. Skipping retraining.")
    exit(1)
else:
    print("Data is fresh. Proceeding with retraining.")

Add this as a step before train.py. This ensures your model retraining automation only runs when it makes sense.

Step 5: Monitor and Rollback

Even with validation, things can go wrong in production. Set up a monitoring step that checks model performance after deployment. You can use MLflow's model registry to roll back:

# Rollback to previous version
mlflow deployments update --model-uri "models:/fraud-detection-model/3" --target $DEPLOY_HOST

This can be triggered manually or via a webhook if your monitoring system (e.g., Prometheus + Grafana) detects degradation.

Production Considerations

This pipeline works great for small to medium projects. For larger scale, consider:

• Parallel training: Use matrix strategies in GitHub Actions to train multiple models simultaneously
• GPU support: GitHub Actions doesn't offer GPUs for free. Use self-hosted runners or a separate CI service for GPU workloads
• Model registry cleanup: Automate deletion of old versions (MLflow supports lifecycle stages: Staging, Production, Archived)
• Security: Use GitHub Actions OIDC tokens to authenticate to cloud providers instead of long-lived secrets

If you're building a full production ML infrastructure—including feature stores, Kubeflow pipelines, A/B testing, and cost optimization—the Production ML (MLOps) course on asibiont.com covers all of this in depth.

Real-World Example: Fraud Detection at Scale

A real e-commerce company implemented this exact pipeline. Before automation, model retraining took two engineers three days. After:

• Retraining runs every Monday at 6 AM automatically
• Model quality is validated against a holdout set
• Best model is deployed to staging, then promoted to production after a 24-hour shadow test
• Rollback happens in under 5 minutes

They reduced time-to-deployment from days to minutes, and false positives dropped by 40% because models were always fresh.

Conclusion

Automating model retraining and deployment with GitHub Actions and MLflow isn't just a nice-to-have—it's a necessity for any ML team that wants to keep models accurate and responsive to change. The pipeline we built is free, open source, and production-ready with minimal overhead.

Start small: set up retraining on a weekly schedule, add validation, then expand to data freshness checks and rollback. Each step reduces manual work and increases reliability.

For a comprehensive deep dive—including Kubeflow, A/B testing, data drift monitoring, and hyperparameter tuning at scale—explore the Production ML (MLOps) course on asibiont.com. It's designed to take you from prototype to production ML infrastructure.

Your models deserve to be as automated as your code.

Ready to build? Fork the example repo, add your MLflow server URL, and trigger your first automated retrain. The future of MLOps is here—and it's free.