You trained a model with 0.95 accuracy on a holdout set. A month later, in production, accuracy drops to 0.72, and nobody notices — because quality metrics aren't being logged. Sound familiar? This is the classic story of the transition from research to production. In this article, we'll break down how Production ML (MLOps) actually works, where standard approaches break down, and how to deploy ML models so they bring value rather than headaches.
Why MLOps is not just "deploying a model"
Production ML is built on a paradigm shift: a model is not an artifact but a service that lives over time. In research, you evaluate a model on a fixed test set. In production, data changes every second: users behave differently, new products appear, the economics change. A model trained on last year's data can become dangerous six months later.
The second key point is reproducibility. If you need to retrain or roll back a model three months from now, you must know what data it was trained on, with what hyperparameters, and with what code. Without this, MLOps doesn't exist.
The third is cost. Training and inference cost money. An unoptimized pipeline can eat through a budget many times larger than the value it delivers.
Breaking down a real task: from model to service
Let's imagine an online store that wants to predict the probability of purchase (CTR) for recommendations. There's a trained model on historical data with 0.85 accuracy. The task is to launch it in production and maintain quality.
Step 1. Feature store. Raw data lives in different places: click logs in S3, user profiles in PostgreSQL, the product catalog in MongoDB. For the model to use features, they need to be collected into a single store with a guarantee of consistency between training and inference. A feature store (for example, Feast or a custom solution) solves this problem: you define features once and use them both in the training pipeline and in the online service.
Step 2. Model serving. The model needs to be wrapped in an API. Options: REST on FastAPI, gRPC for low latency, or specialized solutions like Triton Inference Server. For a CTR model with latency <50 ms, gRPC is often chosen — it's more network-efficient. But if the load is small, you can start with simple REST.
Step 3. A/B test. You can't just replace the old model with a new one — you need to make sure it improves the business metric. We launch an A/B test: 50% of traffic to the old model, 50% to the new one. We look not only at CTR but also at conversion and revenue. It's important to correctly calculate the sample size and test duration, otherwise the result will be random.
Step 4. Monitoring. After launch, you need to watch for data drift. For example, if the average user age has risen by 5 years and the model hasn't seen such data, quality will drop. Monitoring should track the distribution of key features and quality metrics (if feedback is available).
What will go wrong if you skip a step. Without a feature store, you'll get a discrepancy between training and online features (training-serving skew). For example, during training you computed the average order value over 30 days, but in the online service you accidentally took 7 days — accuracy will drop unnoticed. Without an A/B test, you might deploy a model that's worse on business metrics but better on accuracy. Without monitoring, drift will go unnoticed until the losses become obvious.
Where MLOps breaks: failure scenarios
Even a well-designed system can fail. Here are four typical scenarios.
1. Conditions changed: data drift
What goes wrong: the model is trained on pandemic data, when people ordered everything online. After restrictions were lifted, behavior changed, and the model loses accuracy.
How to notice in advance: monitor the distribution of input features (for example, using the Kolmogorov-Smirnov test) and compare it with a reference. If the distance exceeds a threshold — that's a signal.
What to do if it's already too late: urgently retrain the model on fresh data or roll back to the previous version. Ideally, have an automatic retraining pipeline triggered by a trigger.
2. The rule isn't applied: outdated business logic
What goes wrong: the model has a hardcoded rule: if a product costs more than about $100, show it only to users with a high rating. But the business changed its policy — now the threshold is about $50. The model keeps working the old way.
How to notice: regular audit of business rules in the model. It's good if the rules are moved into a config rather than hardcoded.
What to do: update the rule and retrain the model if it affects features.
3. The deadline has passed: model staleness
What goes wrong: a model for predicting customer churn was trained a year ago. In that time, new products and competitors have appeared, and the economics have changed. The model gives incorrect forecasts.
How to notice: set a model expiration date (for example, 6 months) and an automatic retraining reminder.
What to do: retrain on current data. If there's little data, use transfer learning or simplified heuristics.
4. The tool stopped working: infrastructure failure
What goes wrong: the feature store goes down, and the online service can't get features. The model returns an error.
How to notice: monitoring of service availability and alerts.
What to do: have a fallback — for example, cached features or a simplified model. It's also important to provide for graceful degradation.
Paths to mastering MLOps: self-study, courses, practice
You can master MLOps in different ways, and each path has a cost.
Self-study. You read the Kubeflow and MLflow documentation, articles on Medium, watch conference talks. Pros: free, flexible schedule. Cons: no structure, easy to get lost in the weeds, no feedback on your decisions. Time to the first production pipeline — from 6 months to infinity, depending on discipline.
Courses. A structured program, feedback from instructors, analysis of typical mistakes. Pros: faster, there's support, you don't have to find materials yourself. Cons: they cost money, you have to fit the schedule. Time — 2-4 months with intensive work.
Practice at work. The most effective path if you have access to a real project. Pros: real tasks, responsibility. Cons: there isn't always a mentor, you can pick up a lot of bumps and bruises.
An honest trade-off: if you're just starting out and don't have a project to practice on, a course provides the structure and feedback that self-study lacks. If you already work on a team with an MLOps engineer, practice under guidance may be enough.
Who Production ML (MLOps) suits, and who it doesn't
This path is worth choosing if you are:
- A Data Scientist who wants to take models all the way to production, not just build prototypes.
- An ML Engineer who is responsible for infrastructure and wants to systematize their knowledge.
- A developer who is moving into ML and wants to understand the full cycle.
You shouldn't go into MLOps if:
- You're not familiar with the basics of machine learning and don't understand how models are trained. First, it's worth mastering basic ML.
- You're not ready for routine: a lot of time goes into setting up pipelines, monitoring, and debugging infrastructure rather than creative work with models.
- You need quick results without diving into infrastructure — then it's better to focus on data analysis.
Conclusion
Production ML (MLOps) is the discipline of making a model useful in the real world. It requires engineering thinking, attention to detail, and a readiness for failures. If you want to systematically master this field, feedback on your decisions and analysis of real scenarios speed up the process. The Production ML (MLOps) course provides a structured path and practical assignments that help you go through this journey consciously. See also , and Return Money for a Lost Parcel.
You can start here: ML in Production. See also , and Microservices Architecture in Practice.