10 Battle-Tested Prompts for Machine Learning with Scikit-learn, XGBoost, and CatBoost
Introduction
Every data scientist has a set of go-to code snippets that save hours of trial and error. Whether you’re building a quick prototype or deploying a production model, having battle-tested prompts for common ML tasks ensures consistency and speed. This article compiles 10 practical prompts covering the entire workflow — from preprocessing and feature engineering to training, tuning, and evaluating models with three of the most popular frameworks: Scikit-learn, XGBoost, and CatBoost.
Each prompt includes a real Python code example, an explanation of what it does, and a tip extracted from production experience. These snippets are meant to be adapted to your own data and problem. All examples assume you have pandas, numpy, scikit-learn, xgboost, and catboost installed (and optionally shap for interpretability).
1. Automated Preprocessing Pipeline with ColumnTransformer and Pipeline
When to use: You have a mix of numerical and categorical features and want to avoid data leakage during cross-validation.
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import OneHotEncoder, StandardScaler
numeric_features = ['age', 'income', 'credit_score']
categorical_features = ['education', 'occupation']
preprocessor = ColumnTransformer(
transformers=[
('num', Pipeline([
('imputer', SimpleImputer(strategy='median')),
('scaler', StandardScaler())
]), numeric_features),
('cat', Pipeline([
('imputer', SimpleImputer(strategy='constant', fill_value='missing')),
('onehot', OneHotEncoder(handle_unknown='ignore'))
]), categorical_features)
]
)
pipeline = Pipeline(steps=[
('preprocessor', preprocessor),
('classifier', LogisticRegression())
])
Tip: Use ColumnTransformer with remainder='passthrough' if you have features that don’t need transformation. Always set handle_unknown='ignore' in OneHotEncoder to handle unseen categories in production.
2. Feature Selection using SelectKBest and Mutual Information
When to use: You have many features and want to reduce dimensionality based on their relevance to the target.
from sklearn.feature_selection import SelectKBest, mutual_info_classif
selector = SelectKBest(score_func=mutual_info_classif, k=20)
X_selected = selector.fit_transform(X_train, y_train)
# Get selected feature indices
selected_indices = selector.get_support(indices=True)
selected_features = all_features[selected_indices]
Tip: For regression tasks use mutual_info_regression. Mutual information captures non‑linear relationships. Start with k=sqrt(n_features) and increase iteratively.
3. Handling Missing Values with IterativeImputer
When to use: Missing values follow a pattern or you suspect they depend on other features; mean/median imputation would bias the model.
from sklearn.experimental import enable_iterative_imputer
from sklearn.impute import IterativeImputer
from sklearn.ensemble import RandomForestRegressor
imputer = IterativeImputer(
estimator=RandomForestRegressor(n_estimators=10),
max_iter=10,
random_state=0
)
X_imputed = imputer.fit_transform(X)
Tip: IterativeImputer models each feature with missing values as a function of other features. It can be slow on large datasets — consider using ExtraTreesRegressor as a lighter estimator.
4. Training a Gradient Boosting Model with XGBoost (with Early Stopping)
When to use: You need a strong gradient boosting model with built-in regularization and early stopping to prevent overfitting.
import xgboost as xgb
dtrain = xgb.DMatrix(X_train, label=y_train)
dval = xgb.DMatrix(X_val, label=y_val)
params = {
'objective': 'binary:logistic',
'max_depth': 6,
'eta': 0.1,
'subsample': 0.8,
'colsample_bytree': 0.8,
'eval_metric': 'logloss'
}
model = xgb.train(
params,
dtrain,
num_boost_round=1000,
evals=[(dval, 'eval')],
early_stopping_rounds=20,
verbose_eval=50
)
Tip: Use early_stopping_rounds with a separate validation set. The best iteration is stored in model.best_iteration. For large datasets, enable tree_method='hist' for faster training.
5. Hyperparameter Tuning with GridSearchCV and RandomizedSearchCV
When to use: You have a small to medium hyperparameter space and want an exhaustive or randomized search.
from sklearn.model_selection import GridSearchCV, RandomizedSearchCV
from sklearn.ensemble import RandomForestClassifier
from scipy.stats import randint
param_grid = {
'n_estimators': [100, 200],
'max_depth': [5, 10, None],
'min_samples_split': [2, 5]
}
grid_search = GridSearchCV(RandomForestClassifier(random_state=42),
param_grid, cv=5, scoring='f1', n_jobs=-1)
grid_search.fit(X_train, y_train)
print("Best params:", grid_search.best_params_)
# For larger spaces use random search
param_distributions = {
'n_estimators': randint(50, 300),
'max_depth': [3, 5, 7, 10, None]
}
random_search = RandomizedSearchCV(RandomForestClassifier(),
param_distributions,
n_iter=20, cv=3, random_state=42)
random_search.fit(X_train, y_train)
Tip: Use RandomizedSearchCV when you have more than 10 parameters; it’s more efficient. Set n_jobs=-1 to use all CPU cores.
6. Training a Categorical Boosting Model with CatBoost (Handling Categorical Features Natively)
When to use: Your dataset contains many categorical features with high cardinality; CatBoost can use them without manual encoding.
from catboost import CatBoostClassifier
# Identify categorical feature indices
categorical_features_indices = [1, 3, 5] # replace with actual indices
model = CatBoostClassifier(
iterations=500,
learning_rate=0.1,
depth=6,
loss_function='Logloss',
cat_features=categorical_features_indices,
verbose=100
)
model.fit(X_train, y_train, eval_set=(X_val, y_val), early_stopping_rounds=50)
Tip: CatBoost handles missing values and categorical features automatically. Use cat_features as a list of indices or column names. For large datasets, set task_type='GPU' for speed.
7. Model Evaluation with Cross-Validation and Multiple Metrics
When to use: You need a robust estimate of model performance using multiple scoring metrics.
from sklearn.model_selection import cross_validate
from sklearn.ensemble import RandomForestClassifier
import numpy as np
scoring = ['accuracy', 'precision', 'recall', 'f1', 'roc_auc']
scores = cross_validate(RandomForestClassifier(n_estimators=100),
X, y, cv=5,
scoring=scoring,
return_train_score=False)
for metric in scoring:
print(f"{metric}: {np.mean(scores['test_' + metric]):.3f} +/- {np.std(scores['test_' + metric]):.3f}")
Tip: cross_validate returns a dict of scores. Use return_estimator=True to get the fitted models for each fold if you need to inspect them further.
8. Feature Importance Analysis using SHAP and Built-in Importances
When to use: You need to explain model predictions or identify the most influential features.
import shap
from xgboost import XGBClassifier
model = XGBClassifier().fit(X_train, y_train)
# SHAP explainer
shap_explainer = shap.TreeExplainer(model)
shap_values = shap_explainer.shap_values(X_test)
# Summary plot (requires matplotlib)
shap.summary_plot(shap_values, X_test, feature_names=feature_names)
# Built-in importance
importances = model.feature_importances_
indices = np.argsort(importances)[::-1]
for i in range(5):
print(f"{i+1}. {feature_names[indices[i]]}: {importances[indices[i]]:.4f}")
Tip: SHAP provides consistent and theoretically sound feature attributions. For linear models, use shap.LinearExplainer. Always compute SHAP on a representative subset if the dataset is large.
9. Ensemble Learning: Stacking Scikit-learn, XGBoost, and CatBoost
When to use: You want to combine the strengths of different algorithms to improve predictive performance.
from sklearn.ensemble import StackingClassifier
from sklearn.linear_model import LogisticRegression
from xgboost import XGBClassifier
from catboost import CatBoostClassifier
base_learners = [
('rf', RandomForestClassifier(n_estimators=100, random_state=42)),
('xgb', XGBClassifier(n_estimators=100, learning_rate=0.1)),
('cat', CatBoostClassifier(iterations=100, verbose=0))
]
stacking_model = StackingClassifier(
estimators=base_learners,
final_estimator=LogisticRegression()
)
stacking_model.fit(X_train, y_train)
y_pred = stacking_model.predict(X_test)
Tip: Use a simple model like Logistic Regression as the meta-learner. Ensure base learners are trained on different subsets (use cv parameter in StackingClassifier) to avoid overfitting.
10. Saving and Loading Models with joblib and pickle
When to use: You want to persist a trained model for later inference or deployment.
import joblib
# Save
joblib.dump(stacking_model, 'stacked_model.joblib')
# Load
loaded_model = joblib.load('stacked_model.joblib')
# For XGBoost native model
model.save_model('xgb_model.json')
loaded_xgb = xgb.Booster()
loaded_xgb.load_model('xgb_model.json')
# For CatBoost
model.save_model('catboost_model.cbm')
loaded_cat = CatBoostClassifier()
loaded_cat.load_model('catboost_model.cbm')
Tip: Use joblib for Scikit-learn estimators; XGBoost and CatBoost have their own serialization formats (JSON / binary). Always test that the loaded model produces the same predictions as the original.
Conclusion
These 10 prompts cover the core ML workflow using three industry‑standard libraries. Each snippet has been used in real projects — from credit scoring to customer churn prediction — and can be adapted to your specific data and problem. The true power comes from combining them: use the preprocessing pipeline for data cleaning, then try XGBoost or CatBoost with tuning, evaluate with cross‑validation, and finally interpret with SHAP.
Save this article as a reference. For deeper dives, always check the official documentation:
- Scikit-learn: https://scikit-learn.org/stable/user_guide.html
- XGBoost: https://xgboost.readthedocs.io/en/stable/
- CatBoost: https://catboost.ai/en/docs/
Now go ahead and experiment with your own dataset. The best way to learn is to run the code.
Comments