You downloaded a dataset with 50,000 rows, loaded it into Pandas, trained a Random Forest, and got an accuracy of 0.92 on the test set. A week later, the model in production delivers 0.65. Sound familiar? The reason isn't the model — it's the data: missing values, outliers, target leakage, and improper validation. In this article, we'll break down why sample size in machine learning isn't the main thing, and how data cleaning, cross-validation, and overfitting control change the outcome.
Why "more data" isn't always better
There's a persistent myth in Data Science: the more data, the more accurate the model. In practice, it's the opposite. If the data contains noise, missing values, and outliers, increasing the volume only amplifies the error. The model memorizes the garbage and overfits.
Imagine: you're analyzing customer churn at a bank. You have 100,000 records, but in 30% of cases age is missing, and in 5% income is listed in thousands instead of dollars. If you simply drop the missing values, you'll be left with 70,000 rows, but among them are outliers that will skew the mean. The model will train on distorted data and will make mistakes on new customers.
Key idea: data quality matters more than quantity. 500 clean, well-prepared rows often yield a better model than 50,000 dirty ones.
df.info(), df.describe(), and df.duplicated().sum() in Pandas.
A real-world case study: customer churn prediction
Let's consider a typical case: a telecom company wants to predict whether a customer will leave. There's historical data for a year: 10,000 rows, 20 features (age, plan, number of calls, complaints, etc.). The target variable is whether the customer churned (1) or stayed (0).
Step 1. Data cleaning
First, we check for missing values. The "average bill" column has 15% missing. If we simply drop these rows, we'll lose 1,500 records. It's better to fill missing values with the median or predict them with a model. But there's a nuance: missing values may not be random. For example, high-bill customers are more likely not to report income. Then filling with the median will distort the picture.
Data cleaning is not just about removing missing values. It's about analyzing the reasons for their appearance. If a missing value is related to the target variable, it cannot be ignored.
Outliers: the "number of calls" feature has values of 10,000 when the average is 200. These are obvious input errors. They need to be either removed or replaced with the 99th percentile. Leaving them is not an option — the model may give them excessive weight.
Step 2. Eliminating target leakage
One of the most insidious mistakes. The dataset included the feature "number of support requests in the last month." It seems useful. But if a customer has already left, they don't call. The feature indirectly contains information about the target variable. The model will show excellent metrics on the test set, but in reality it will be useless because for new customers we don't know whether they will leave or not, and therefore we don't know the number of requests after churn either.
How to detect leakage: check whether any feature correlates too strongly with the target variable (correlation > 0.9). Or think about whether the feature is available at the time of prediction. If not, remove it.
Step 3. Cross-validation instead of a single train/test split
Many people split data into training and test sets in an 80/20 ratio and call it a day. But with a small data volume (e.g., 1,000 rows), such an estimate is unstable: the test set may randomly turn out to be easy or hard.
Cross-validation solves this problem: data is split into k blocks (usually 5 or 10), the model is trained k times, each time one part serves as the test set. The final metric is the average across all folds. This gives a more reliable estimate and helps spot overfitting.
Example: with 5-fold cross-validation, accuracy might be 0.85, 0.87, 0.84, 0.86, 0.88. The average is 0.86, with a small spread. If the spread is large (0.7–0.95), it means the model is unstable and the data is heterogeneous.
cross_val_score from scikit-learn. For classification, use StratifiedKFold to preserve class balance in each fold.
Step 4. Overfitting control
Overfitting is when a model shows excellent results on the training set but poor results on new data. Causes: too complex a model, too little data, too much noise.
How to combat it:
- Regularization (L1, L2) for linear models.
- Limiting tree depth and minimum samples per leaf.
- Early stopping for gradient boosting.
- Learning curves: plot the metric against the training set size. If the curve plateaus, adding data won't help — you need to change the model or features.
Step 5. Feature engineering
Creating new features can yield more than increasing the dataset. For example, from the connection date you can extract customer tenure, from the sum of top-ups — the average monthly bill. Well-designed features improve accuracy even on small data.
Non-obvious conclusion: sometimes removing one or two features improves the model. If a feature is noisy or highly correlated with another, it only gets in the way.
How much data do you really need?
The answer depends on the task and model complexity. For simple linear regression, 100–200 rows per feature may suffice. For deep learning, you need thousands. But the key factor is not quantity — it's representativeness and cleanliness.
Check for class imbalance. If in a churn task 95% of customers stayed and 5% left, the model can simply predict "stayed" and have an accuracy of 0.95, but fail to identify churners. In such cases, use precision, recall, F1 metrics and balancing methods (oversampling, undersampling, SMOTE).
Rule: first clean the data, eliminate leakage, set up cross-validation. Only then think about expanding the sample.
Failure modes: where the mechanism breaks
Even experienced specialists make mistakes. Here are three typical scenarios where the model behaves unpredictably.
1. Target leakage
What goes wrong: the model shows an accuracy of 0.99 on cross-validation, but 0.6 in production. The reason: a column that is a consequence of the target variable got into the features. For example, "contract closure date" for a churn prediction task.
How to spot it in advance: check the correlation of each feature with the target variable. If it's close to 1 or -1, that's suspicious. It's also useful to train a model on just one suspicious feature — if the metric is high, the feature contains leakage.
What to do: remove such a feature or replace it with historical data available at the time of prediction.
2. Overfitting to noise
What goes wrong: a model with 1,000 trees perfectly memorizes the training set (accuracy 1.0), but on the test set — 0.7. This is classic overfitting.
How to spot it in advance: build learning and validation curves. If the training metric rises while the validation metric falls, the model is overfitting.
What to do: simplify the model, add regularization, increase the data volume (if possible), or reduce the number of features.
3. Sample bias
What goes wrong: the model was trained on data from last year, but in the new year the plans changed. The features are distributed differently, and the model makes mistakes.
How to spot it in advance: compare the feature distributions in the training and test sets (e.g., using the Kolmogorov-Smirnov test). If they differ, the model may perform incorrectly.
What to do: update the training set, use domain adaptation, or retrain the model on new data.
Paths to mastering Data Science: self-taught vs structured
If you want to understand the topic, there are several paths.
Self-taught: free courses, documentation, Kaggle. Pros — cheap, cons — no feedback, easy to miss important details (e.g., data leakage). Time — from 6 months to infinity.
Courses with a mentor: a structured program, homework checks, error analysis. Costs money but saves time and nerves. For example, Data Science from Scratch includes practical projects and feedback on your code.
Practice on real tasks: the most effective way, but requires basic knowledge. You can start with Kaggle competitions.
Trade-off: self-study is cheaper, but the risk of getting stuck on mistakes that no one will correct is higher. Courses provide structure and feedback, but cost money and time.
Who Data Science suits and who it doesn't
Data Science is not for everyone. If you don't like analyzing data, finding patterns, and constantly learning, it will be hard. Also, you shouldn't go into DS if you expect quick results without effort.
But if you enjoy digging into numbers, are ready to tolerate uncertainty, and want to influence business decisions — this is your path.
Non-obvious conclusion: often people who come into DS are not mathematicians but those with domain expertise (finance, medicine, marketing). They understand the data better and can spot leakage or bias sooner than a pure programmer.
Checklist before training a model
- Check for missing values and outliers. Fill or remove them consciously.
- Make sure there is no target leakage. Features must be available at the time of prediction.
- Use cross-validation, not a single train/test split.
- Check for class imbalance. Choose the right metrics.
- Build learning curves to control overfitting.
- Do feature engineering: create meaningful features.
If you want to go through this path systematically and get feedback on each step, check out the course Data Science from Scratch. It covers real projects — from data cleaning to building reproducible pipelines.