15 Prompts for Machine Learning: Scikit-learn, XGBoost, and CatBoost

15 Prompts for Machine Learning: Scikit-learn, XGBoost, and CatBoost

Machine learning (ML) is no longer a niche specialty—it’s a core competency for data scientists, analysts, and even software engineers. Yet, one of the biggest bottlenecks in any ML project is not the algorithm itself, but the prompting: how do you ask the right questions, preprocess data correctly, and tune models efficiently? This article provides 15 ready-to-use, battle-tested prompts for the three most popular ML frameworks: Scikit-learn, XGBoost, and CatBoost. Each prompt is designed to save you hours of trial and error, backed by official documentation and real-world use cases.

Whether you’re a beginner building your first classification model or an expert optimizing a gradient boosting pipeline, these prompts will accelerate your workflow. Let’s dive in.

Why Prompts Matter in Machine Learning

In ML, a “prompt” is more than just a command—it’s a structured instruction that guides the model or the tool to produce a desired outcome. For Scikit-learn, XGBoost, and CatBoost, prompts often translate into code snippets, configuration patterns, or decision trees for hyperparameter tuning. According to the official Scikit-learn documentation (scikit-learn.org/stable/documentation.html), proper preprocessing and pipeline design can reduce model error by up to 30% compared to ad-hoc approaches. Similarly, XGBoost’s documentation (xgboost.readthedocs.io) emphasizes that parameter tuning can be the difference between a mediocre and a state-of-the-art model. CatBoost’s official guide (catboost.ai/docs) highlights its native support for categorical features, which eliminates the need for manual one-hot encoding—a common source of errors.

This article is structured as a cheat sheet. Each prompt includes: (1) a specific task, (2) an explanation of why it works, and (3) a copy-paste code example. Use them as building blocks for your own projects.

Prompts for Scikit-learn

Scikit-learn is the Swiss Army knife of ML in Python. Its unified API, extensive documentation, and robust implementations make it ideal for classical algorithms (regression, classification, clustering, dimensionality reduction). Below are five prompts that cover preprocessing, model selection, and evaluation.

Prompt 1: Automated Preprocessing with ColumnTransformer

Task: Apply different preprocessing pipelines to numeric and categorical columns in one step.

Explanation: Scikit-learn’s ColumnTransformer lets you apply separate transformations to subsets of features, which is critical when dealing with mixed data types (e.g., numeric values that need scaling, and categorical values that need encoding). This avoids the common mistake of scaling categorical variables or one-hot encoding numeric ones.

Usage example:

from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.pipeline import Pipeline

# Assume 'num_features' and 'cat_features' are lists of column names
preprocessor = ColumnTransformer(
    transformers=[
        ('num', StandardScaler(), num_features),
        ('cat', OneHotEncoder(drop='first'), cat_features)
    ])
pipeline = Pipeline(steps=[('preprocessor', preprocessor),
                           ('classifier', LogisticRegression())])

This prompt is especially useful when you have a mix of ages, incomes (numeric) and cities, genders (categorical). It’s a pattern used in many Kaggle competition winners.

Prompt 2: Grid Search for Hyperparameter Tuning

Task: Find the best hyperparameters for a Random Forest classifier using cross-validation.

Explanation: Grid search exhaustively evaluates all combinations of specified parameters. While computationally expensive, it’s the gold standard for small to medium datasets. The prompt below uses GridSearchCV with 5-fold cross-validation to avoid overfitting.

Usage example:

from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import GridSearchCV

param_grid = {
    'n_estimators': [50, 100, 200],
    'max_depth': [None, 10, 20],
    'min_samples_split': [2, 5, 10]
}
grid_search = GridSearchCV(RandomForestClassifier(random_state=42),
                           param_grid, cv=5, scoring='f1', n_jobs=-1)
grid_search.fit(X_train, y_train)
print('Best params:', grid_search.best_params_)

According to a 2024 study by the University of Cambridge (published in Journal of Machine Learning Research), grid search with 5-fold CV reduces generalization error by an average of 12% compared to manual tuning.

Prompt 3: Feature Selection with Recursive Feature Elimination

Task: Identify the most important features for a linear SVM.

Explanation: RFE recursively removes the least important features based on model coefficients. This is crucial when you have hundreds of features and want to reduce dimensionality without losing predictive power.

Usage example:

from sklearn.svm import SVC
from sklearn.feature_selection import RFE

estimator = SVC(kernel='linear')
selector = RFE(estimator, n_features_to_select=10, step=1)
selector.fit(X_train, y_train)
print('Selected features:', selector.support_)

Note: RFE can be slow for large datasets. For faster alternatives, consider SelectFromModel.

Prompt 4: Model Evaluation with Cross-Validation Metrics

Task: Compute multiple metrics across cross-validation folds.

Explanation: Scikit-learn’s cross_validate function returns scores for multiple metrics, allowing you to assess precision, recall, F1, and accuracy simultaneously. This is essential for imbalanced datasets where accuracy alone is misleading.

Usage example:

from sklearn.model_selection import cross_validate
from sklearn.ensemble import GradientBoostingClassifier

model = GradientBoostingClassifier()
scores = cross_validate(model, X, y, cv=5,
                        scoring=['precision', 'recall', 'f1'],
                        return_train_score=False)
for metric in ['test_precision', 'test_recall', 'test_f1']:
    print(f'{metric}: {scores[metric].mean():.3f} (+/- {scores[metric].std():.3f})')

This pattern is recommended by the official Scikit-learn documentation for robust model validation.

Prompt 5: Imbalanced Dataset Handling with SMOTE

Task: Generate synthetic samples for the minority class using SMOTE.

Explanation: SMOTE (Synthetic Minority Over-sampling Technique) creates new samples by interpolating between existing minority class instances. It’s a standard technique for fraud detection, medical diagnosis, and any scenario with class imbalance.

Usage example:

from imblearn.over_sampling import SMOTE
from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)
smote = SMOTE(random_state=42)
X_train_resampled, y_train_resampled = smote.fit_resample(X_train, y_train)

Note: SMOTE is from the imbalanced-learn library, which integrates seamlessly with Scikit-learn.

Prompts for XGBoost

XGBoost is a gradient boosting framework known for its speed and performance, especially in structured/tabular data competitions. It won the Higgs Boson Machine Learning Challenge in 2014 and remains a top choice in 2026. The following prompts focus on its unique features: early stopping, custom objective functions, and GPU acceleration.

Prompt 6: Early Stopping to Prevent Overfitting

Task: Train an XGBoost classifier with early stopping based on validation set performance.

Explanation: Early stopping halts training when the validation metric stops improving for a specified number of rounds. This prevents overfitting and saves computational resources. XGBoost’s native eval_set parameter makes this straightforward.

Usage example:

import xgboost as xgb

model = xgb.XGBClassifier(n_estimators=1000, early_stopping_rounds=10, eval_metric='logloss')
model.fit(X_train, y_train, eval_set=[(X_test, y_test)], verbose=False)

According to XGBoost’s official documentation (xgboost.readthedocs.io), early stopping can reduce overfitting by up to 40% compared to training without it.

Prompt 7: Custom Objective Function for Regression

Task: Implement a custom objective function (e.g., Huber loss) for robust regression.

Explanation: XGBoost allows you to define custom objective functions and evaluation metrics. This is useful when standard loss functions (e.g., MSE) are not appropriate—for example, when your data contains outliers.

Usage example:

import numpy as np

def huber_loss(predt, dtrain):
    labels = dtrain.get_label()
    delta = 1.0
    residual = predt - labels
    grad = np.where(np.abs(residual) <= delta, residual, delta * np.sign(residual))
    hess = np.where(np.abs(residual) <= delta, 1.0, 0.0)
    return grad, hess

model = xgb.train({'objective': huber_loss, 'eval_metric': 'mae'},
                  dtrain, num_boost_round=100)

This pattern is documented in the XGBoost tutorials and is widely used in financial forecasting.

Prompt 8: GPU Acceleration for Large Datasets

Task: Train XGBoost on a large dataset using GPU to speed up training.

Explanation: XGBoost supports GPU training via the tree_method='gpu_hist' parameter. On modern NVIDIA GPUs (e.g., A100, H100), this can accelerate training by 5–10x compared to CPU, according to benchmark tests by the XGBoost team.

Usage example:

model = xgb.XGBClassifier(tree_method='gpu_hist', predictor='gpu_predictor')
model.fit(X_train, y_train)

Note: Ensure you have installed XGBoost with GPU support (pip install xgboost includes it by default since version 1.7).

Prompt 9: Feature Importance Plotting

Task: Visualize the most important features after training.

Explanation: XGBoost provides built-in feature importance metrics (weight, gain, cover). Plotting them helps with interpretability and feature selection.

Usage example:

import matplotlib.pyplot as plt

model = xgb.XGBClassifier()
model.fit(X_train, y_train)
xgb.plot_importance(model, importance_type='gain', max_num_features=10)
plt.show()

This is a quick way to explain model decisions to stakeholders.

Prompt 10: Handling Missing Values with Native Support

Task: Train an XGBoost model with missing values automatically handled.

Explanation: XGBoost natively handles missing values by learning the best direction (left or right) to send missing instances during tree construction. No imputation required.

Usage example:

# X_train contains np.nan values
model = xgb.XGBClassifier(missing=np.nan)  # default is np.nan
model.fit(X_train, y_train)

This is a major time-saver compared to Scikit-learn, which requires imputation first.

Prompts for CatBoost

CatBoost (Categorical Boosting) is a gradient boosting library developed by Yandex. Its standout feature is native support for categorical features without manual encoding, which reduces preprocessing steps and often improves accuracy. The following prompts cover its unique capabilities.

Prompt 11: Automatic Categorical Feature Handling

Task: Train a CatBoost classifier without manually encoding categorical columns.

Explanation: CatBoost automatically converts categorical features using a combination of one-hot encoding and target encoding (ordered TS). You just need to specify which columns are categorical.

Usage example:

from catboost import CatBoostClassifier

cat_features = [0, 2, 5]  # indices of categorical columns
model = CatBoostClassifier(iterations=500, cat_features=cat_features, verbose=False)
model.fit(X_train, y_train)

According to CatBoost’s official benchmark (catboost.ai/docs/concepts/speed-acc.html), this approach reduces preprocessing time by up to 80% compared to manual encoding, while maintaining or improving accuracy.

Prompt 12: Fast Training with Learning Rate Scheduling

Task: Use learning rate scheduling to speed up convergence.

Explanation: CatBoost supports adaptive learning rate schedules, such as exponential decay or cosine annealing. This helps escape local minima and reduces the number of iterations needed.

Usage example:

model = CatBoostClassifier(iterations=1000, learning_rate=0.03,
                           lr_schedule='cosine', verbose=100)
model.fit(X_train, y_train)

This is particularly useful for large datasets where each iteration is expensive.

Prompt 13: Model Interpretation with SHAP Values

Task: Explain predictions using SHAP (SHapley Additive exPlanations) values.

Explanation: CatBoost has built-in SHAP support, which provides consistent and mathematically grounded feature attribution. This is essential for regulatory compliance in finance and healthcare.

Usage example:

import shap

model = CatBoostClassifier(iterations=500, verbose=False)
model.fit(X_train, y_train)
shap_values = model.get_feature_importance(data=Pool(X_train, y_train), type='ShapValues')
shap.summary_plot(shap_values, X_train)

Note: CatBoost’s SHAP implementation is faster than the generic shap library, as documented in their official repository.

Prompt 14: Cross-Validation Without Leakage

Task: Perform cross-validation with CatBoost’s built-in CV function, ensuring no data leakage from categorical features.

Explanation: CatBoost’s cv function automatically handles categorical features correctly during cross-validation, unlike manual loops that can cause target leakage.

Usage example:

from catboost import cv

params = {'iterations': 100, 'loss_function': 'Logloss', 'random_seed': 42}
cv_data = cv(Pool(X_train, y_train, cat_features=cat_features), params, fold_count=5)
print('Best iteration:', cv_data['iterations'][cv_data['test-Logloss-mean'].idxmin()])

This is a best practice recommended by Yandex’s ML engineers.

Prompt 15: GPU Training for CatBoost

Task: Train CatBoost on GPU for large-scale datasets.

Explanation: CatBoost supports GPU acceleration via the task_type='GPU' parameter. It can handle datasets with millions of rows efficiently.

Usage example:

model = CatBoostClassifier(iterations=1000, task_type='GPU', devices='0:1', verbose=False)
model.fit(X_train, y_train)

CatBoost’s GPU implementation is highly optimized, with benchmarks showing up to 20x speedup on large datasets (catboost.ai/docs/features/gpu.html).

Conclusion

These 15 prompts form a practical toolkit for any ML practitioner working with Scikit-learn, XGBoost, or CatBoost. From preprocessing and feature selection to advanced tuning and interpretation, each prompt is designed to be immediately actionable. Remember that the best model is often a blend of these frameworks—for instance, using Scikit-learn for preprocessing, XGBoost for feature importance, and CatBoost for final training on mixed data types. Keep experimenting, and always validate your results with cross-validation.

For a deeper dive into integrating these tools into production ML pipelines, explore resources like the official documentation links provided throughout this article. Happy modeling!