From Experiment to Business Result: Why ML Production Is a Challenge
Every Data Scientist knows the feeling: the model shows excellent metrics in Jupyter Notebook, AUC — 0.98, F1-score is pleasing to the eye. But as soon as it comes to launching in the real world, problems begin. Inference speed drops, predictions "drift" on new data, and the infrastructure can't handle the load. ML production is not just model deployment, but a whole set of engineering practices that turn a raw experiment into a stable API service. In this article, we'll break down the key stages: from building an ML pipeline to A/B testing and monitoring.
Stage 1. Designing the ML Pipeline: From Raw Data to Predictions
An ML pipeline is a sequence of steps that automates data preparation, training, validation, and model deployment. Without it, production turns into chaos. The main components of the pipeline:
- Feature engineering: transforming raw logs, texts, or images into features. For example, for a credit scoring model, you need to aggregate a client's transactions over the last 30 days.
- Training and validation: fixing hyperparameters, data versions, and code. Use DVC or MLflow to track experiments.
- Testing: checking the model on a holdout set, stress tests on synthetic data.
- Packaging: containerizing the model (Docker) + serializing it to ONNX or pickle format.
Practical example: in a product recommendation startup, we used Airflow to orchestrate the pipeline. Every night, a DAG ran that collected the day's data, updated features, and retrained the model. This reduced manual deployment time from 4 hours to 15 minutes.
Stage 2. Model Deployment: From Container to REST API
Model deployment is not just copying a file to a server. You need to ensure scalability, fault tolerance, and low latency. Popular approaches:
- FastAPI + Docker: a lightweight web framework with automatic Swagger documentation. Example code:
from fastapi import FastAPI
import joblib
app = FastAPI()
model = joblib.load('model.pkl')
@app.post('/predict')
async def predict(features: dict):
pred = model.predict([list(features.values())])
return {'prediction': pred.tolist()}
- Kubernetes: for container orchestration. Auto-scaling under load, rolling updates without downtime.
- Serverless: AWS Lambda or Google Cloud Run — pay only for invocations. Suitable for models with infrequent requests.
Important: cache frequent requests (Redis) and use batching. If the model receives 10 requests per second, combine them into a batch of 32 — this reduces latency by 40%.
Stage 3. Monitoring: How Not to Miss Data Drift
After model deployment, the hardest part begins — maintenance. Real-world data changes: new product categories appear, user behavior shifts. This is called data drift. Without monitoring, quality metrics will drop unnoticed.
| Metric | What It Tracks | Tools |
|---|---|---|
| Accuracy/Precision/Recall | Prediction quality | Evidently AI, WhyLabs |
| PSI (Population Stability Index) | Change in feature distribution | Scipy, custom metric |
| P50/P99 latency | API response speed | Prometheus + Grafana |
| Error rate | Share of 4xx/5xx errors | ELK Stack (Elasticsearch, Logstash, Kibana) |
Tip: set up alerts in Telegram/Slack. If PSI > 0.2 — it's a signal to retrain the model. Store prediction logs in ClickHouse for retrospective analysis.
Stage 4. A/B Testing: How to Compare Old and New Models
Replacing a model in production without an A/B test is risky. Even if offline metrics are better, online performance might be worse. A/B testing scheme in ML:
- Split traffic into two groups: control (old model) and experiment (new model).
- Use feature flags (LaunchDarkly) or a request balancer (Nginx).
- Collect business metrics: conversion, CTR, revenue. Statistical significance (
Comments