12 Prompts for Machine Learning: Scikit-learn, XGBoost, CatBoost

12 Prompts for Machine Learning: Scikit-learn, XGBoost, CatBoost

Building a robust machine learning pipeline requires more than just fitting a model. Data preprocessing, feature engineering, hyperparameter tuning, and model evaluation are equally critical. This guide provides 12 ready-to-use prompts for interacting with AI assistants on tasks involving Scikit-learn, XGBoost, and CatBoost. Each prompt includes a clear description, a practical example, and executable code. Whether you are a beginner or a seasoned practitioner, these prompts will save you time and help you avoid common pitfalls.

Why These Prompts Matter

According to the official Scikit-learn documentation (https://scikit-learn.org/stable/), proper preprocessing can improve model accuracy by up to 30% in many real-world datasets. XGBoost and CatBoost are among the top-performing gradient boosting frameworks, consistently winning Kaggle competitions. However, their power comes with complexity: tuning parameters like learning_rate, max_depth, or cat_features can make or break a project. The prompts below cover the entire workflow — from cleaning data to saving the final model — so you can focus on solving the business problem.

Prompt 1: Data Preprocessing with Scikit-learn

Task: Clean and prepare a raw dataset for machine learning.

Prompt:
"I have a CSV file with missing values in numeric columns and one categorical column with 5 unique values. Write a Python script using Scikit-learn's SimpleImputer and OneHotEncoder inside a ColumnTransformer to impute missing numeric values with the median and one-hot encode the categorical column. Use train_test_split with a random state of 42."

Example Usage:

import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import OneHotEncoder
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline

# Sample data
df = pd.DataFrame({
    'age': [25, 30, None, 45, 35],
    'income': [50000, 60000, 55000, None, 48000],
    'city': ['NYC', 'LA', 'NYC', 'Chicago', 'LA']
})
y = [0, 1, 0, 1, 0]

X_train, X_test, y_train, y_test = train_test_split(
    df, y, test_size=0.2, random_state=42
)

numeric_features = ['age', 'income']
categorical_features = ['city']

preprocessor = ColumnTransformer(
    transformers=[
        ('num', SimpleImputer(strategy='median'), numeric_features),
        ('cat', OneHotEncoder(), categorical_features)
    ]
)

X_train_processed = preprocessor.fit_transform(X_train)
X_test_processed = preprocessor.transform(X_test)
print('Shape after preprocessing:', X_train_processed.shape)

Explanation: This prompt creates a reusable preprocessing pipeline that handles missing values and categorical encoding automatically. Using ColumnTransformer is best practice because it prevents data leakage and keeps the code modular.

Prompt 2: Feature Scaling for SVM and KNN

Task: Scale features for distance-based algorithms.

Prompt:
"I'm building a K-Nearest Neighbors classifier. Write a script that uses StandardScaler from Scikit-learn to scale numeric features. Show how fit_transform works on training data and transform on test data to avoid data leakage."

Example Usage:

from sklearn.preprocessing import StandardScaler
from sklearn.neighbors import KNeighborsClassifier
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline

X, y = load_iris(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)

pipeline = make_pipeline(StandardScaler(), KNeighborsClassifier(n_neighbors=3))
pipeline.fit(X_train, y_train)
accuracy = pipeline.score(X_test, y_test)
print(f'Test accuracy: {accuracy:.2f}')

Explanation: StandardScaler centers the data (mean = 0, std = 1). This is crucial for algorithms that rely on distances (KNN, SVM, PCA). The pipeline ensures scaling is applied consistently.

Prompt 3: Train a Random Forest Classifier

Task: Build and evaluate a Random Forest model for classification.

Prompt:
"Using Scikit-learn's RandomForestClassifier, train a model on the Titanic dataset (available via Seaborn). Use 100 trees, max depth of 5, and evaluate with cross-validation (5-fold). Print mean accuracy and standard deviation."

Example Usage:

import seaborn as sns
import pandas as pd
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import cross_val_score
from sklearn.preprocessing import LabelEncoder

# Load data
df = sns.load_dataset('titanic')
df = df.dropna(subset=['age', 'embarked'])
X = df[['pclass', 'age', 'fare']].copy()
y = df['survived']

# Encode categorical
le = LabelEncoder()
X['embarked'] = le.fit_transform(df['embarked'])

model = RandomForestClassifier(n_estimators=100, max_depth=5, random_state=42)
scores = cross_val_score(model, X, y, cv=5)
print(f'Accuracy: {scores.mean():.3f} +/- {scores.std():.3f}')

Explanation: Random Forest is robust and handles non-linear relationships. Cross-validation gives a more reliable estimate of performance than a single train-test split.

Prompt 4: Hyperparameter Tuning with GridSearchCV

Task: Find optimal hyperparameters for a model.

Prompt:
"Use GridSearchCV to tune n_estimators (50, 100, 200) and max_depth (3, 5, 7) for a Random Forest classifier on the Breast Cancer dataset. Use 3-fold cross-validation and roc_auc as the scoring metric. Print the best parameters and best score."

Example Usage:

from sklearn.datasets import load_breast_cancer
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import GridSearchCV

X, y = load_breast_cancer(return_X_y=True)

param_grid = {
    'n_estimators': [50, 100, 200],
    'max_depth': [3, 5, 7]
}

grid = GridSearchCV(
    RandomForestClassifier(random_state=42),
    param_grid,
    cv=3,
    scoring='roc_auc'
)
grid.fit(X, y)
print(f'Best params: {grid.best_params_}')
print(f'Best AUC: {grid.best_score_:.3f}')

Explanation: GridSearchCV exhaustively searches the parameter space. Using roc_auc is appropriate for binary classification with imbalanced classes. Always fit on the entire training set after tuning.

Prompt 5: Train an XGBoost Classifier

Task: Build a gradient boosting model with XGBoost.

Prompt:
"Install XGBoost and train a classifier on the Pima Indians Diabetes dataset. Use max_depth=4, learning_rate=0.1, and n_estimators=100. Evaluate with accuracy and confusion matrix."

Example Usage:

import xgboost as xgb
from sklearn.datasets import load_diabetes  # not exactly, use a local CSV
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score, confusion_matrix
import pandas as pd

# Load dataset (replace with actual path)
url = 'https://raw.githubusercontent.com/jbrownlee/Datasets/master/pima-indians-diabetes.data.csv'
columns = ['preg', 'plas', 'pres', 'skin', 'test', 'mass', 'pedi', 'age', 'class']
df = pd.read_csv(url, names=columns)
X = df.drop('class', axis=1)
y = df['class']

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

model = xgb.XGBClassifier(
    max_depth=4,
    learning_rate=0.1,
    n_estimators=100,
    use_label_encoder=False,
    eval_metric='logloss'
)
model.fit(X_train, y_train)
y_pred = model.predict(X_test)

print(f'Accuracy: {accuracy_score(y_test, y_pred):.2f}')
print('Confusion matrix:')
print(confusion_matrix(y_test, y_pred))

Explanation: XGBoost often outperforms Random Forest on tabular data. The use_label_encoder=False avoids a deprecation warning. The confusion matrix helps understand false positives and false negatives.

Prompt 6: Feature Importance with XGBoost

Task: Visualize which features drive predictions.

Prompt:
"After training an XGBoost model, plot the top 10 feature importances using plot_importance. Explain why some features have higher importance."

Example Usage:

import matplotlib.pyplot as plt
from xgboost import plot_importance

# Assuming model is already trained (from previous prompt)
plot_importance(model, max_num_features=10, importance_type='weight')
plt.show()

# Print numeric values
importance = model.get_booster().get_score(importance_type='weight')
print('Feature importance (weight):', importance)

Explanation: Feature importance helps in model interpretation and feature selection. XGBoost provides three types: weight (number of times used), gain (average gain when used), and cover (average coverage). Weight is the most intuitive.

Prompt 7: Train a CatBoost Classifier

Task: Use CatBoost for datasets with many categorical features.

Prompt:
"Train a CatBoost classifier on the Adult Income dataset (predict >50K income). Use cat_features parameter to specify categorical columns. Set iterations=100, learning_rate=0.1, and verbose=50. Evaluate with ROC-AUC."

Example Usage:

from catboost import CatBoostClassifier
from sklearn.model_selection import train_test_split
from sklearn.metrics import roc_auc_score
import pandas as pd

# Load data
url = 'https://archive.ics.uci.edu/ml/machine-learning-databases/adult/adult.data'
columns = ['age', 'workclass', 'fnlwgt', 'education', 'education-num', 'marital-status',
           'occupation', 'relationship', 'race', 'sex', 'capital-gain', 'capital-loss',
           'hours-per-week', 'native-country', 'income']
df = pd.read_csv(url, names=columns, na_values=' ?', skipinitialspace=True)
df = df.dropna()

cat_features = ['workclass', 'education', 'marital-status', 'occupation',
                'relationship', 'race', 'sex', 'native-country']
X = df.drop('income', axis=1)
y = (df['income'] == '>50K').astype(int)

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

model = CatBoostClassifier(
    iterations=100,
    learning_rate=0.1,
    cat_features=cat_features,
    verbose=50,
    random_seed=42
)
model.fit(X_train, y_train)
y_pred_proba = model.predict_proba(X_test)[:, 1]
print(f'ROC-AUC: {roc_auc_score(y_test, y_pred_proba):.3f}')

Explanation: CatBoost handles categorical features natively without one-hot encoding, which reduces memory and improves speed. It also uses ordered boosting to reduce overfitting.

Prompt 8: CatBoost with Custom Loss Function

Task: Use a custom loss function for imbalanced classification.

Prompt:
"In CatBoost, implement a custom loss function that penalizes false negatives more heavily (e.g., weight = 5 for positive class). Use scale_pos_weight parameter and show the effect on recall."

Example Usage:

from catboost import CatBoostClassifier
from sklearn.metrics import recall_score
# Continuing from previous dataset
model_weighted = CatBoostClassifier(
    iterations=100,
    scale_pos_weight=5,  # positive class weight
    cat_features=cat_features,
    verbose=0,
    random_seed=42
)
model_weighted.fit(X_train, y_train)
y_pred_weighted = model_weighted.predict(X_test)
recall_weighted = recall_score(y_test, y_pred_weighted)
print(f'Recall with weight 5: {recall_weighted:.3f}')

Explanation: Imbalanced datasets (e.g., fraud detection) require special handling. scale_pos_weight adjusts the loss function to focus on the minority class. The recall typically increases, but precision may drop.

Prompt 9: Model Persistence with Joblib

Task: Save and load trained models.

Prompt:
"After training an XGBoost model, save it using joblib.dump and load it back. Also save the preprocessing pipeline. Show how to make predictions with the loaded model."

Example Usage:

import joblib
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from xgboost import XGBClassifier
from sklearn.datasets import load_iris

X, y = load_iris(return_X_y=True)
pipeline = Pipeline([
    ('scaler', StandardScaler()),
    ('xgb', XGBClassifier(n_estimators=50, max_depth=3, use_label_encoder=False, eval_metric='logloss'))
])
pipeline.fit(X, y)

# Save
joblib.dump(pipeline, 'iris_pipeline.pkl')

# Load
loaded_pipeline = joblib.load('iris_pipeline.pkl')
new_data = [[5.1, 3.5, 1.4, 0.2]]
prediction = loaded_pipeline.predict(new_data)
print(f'Predicted class: {prediction[0]}')

Explanation: joblib is more efficient than pickle for large NumPy arrays. Always save the entire pipeline to ensure consistent preprocessing.

Prompt 10: Cross-Validation with Custom Folds

Task: Implement stratified k-fold cross-validation manually.

Prompt:
"Use StratifiedKFold from Scikit-learn to perform 5-fold cross-validation on an XGBoost model. For each fold, print the fold number, train and test AUC. Calculate the mean and standard deviation of AUC across folds."

Example Usage:

from sklearn.model_selection import StratifiedKFold
from xgboost import XGBClassifier
from sklearn.metrics import roc_auc_score
import numpy as np

# Assume X, y are loaded
skf = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
auc_scores = []

for fold, (train_idx, test_idx) in enumerate(skf.split(X, y)):
    X_train_fold, X_test_fold = X.iloc[train_idx], X.iloc[test_idx]
    y_train_fold, y_test_fold = y.iloc[train_idx], y.iloc[test_idx]

    model = XGBClassifier(n_estimators=50, max_depth=3, use_label_encoder=False, eval_metric='logloss')
    model.fit(X_train_fold, y_train_fold)
    y_pred_proba = model.predict_proba(X_test_fold)[:, 1]
    auc = roc_auc_score(y_test_fold, y_pred_proba)
    auc_scores.append(auc)
    print(f'Fold {fold+1}: AUC = {auc:.3f}')

print(f'Mean AUC: {np.mean(auc_scores):.3f} (+/- {np.std(auc_scores):.3f})')

Explanation: StratifiedKFold preserves the class distribution in each fold, which is crucial for imbalanced datasets. Manual loops give you fine-grained control.

Prompt 11: Handling Missing Values in XGBoost

Task: Understand how XGBoost handles missing data internally.

Prompt:
"Explain how XGBoost learns the best direction for missing values during training. Show an example where missing values are left as NaN in the dataframe, and XGBoost handles them without imputation."

Example Usage:

import pandas as pd
import numpy as np
from xgboost import XGBClassifier

# Create data with missing values
np.random.seed(42)
X = pd.DataFrame({
    'feature1': [1.0, 2.0, np.nan, 4.0, 5.0],
    'feature2': [10.0, np.nan, 30.0, 40.0, 50.0]
})
y = [0, 1, 0, 1, 0]

model = XGBClassifier(n_estimators=10, max_depth=2, use_label_encoder=False, eval_metric='logloss')
model.fit(X, y)
predictions = model.predict(X)
print('Predictions with missing values:', predictions)

Explanation: XGBoost has a built-in sparsity-aware algorithm that learns the optimal default direction for missing values. This can outperform imputation in many cases, especially when missingness has a pattern.

Prompt 12: CatBoost with Text Features

Task: Use CatBoost's built-in text feature support.

Prompt:
"CatBoost can handle text columns directly. Train a CatBoost classifier on a dataset with a text column (e.g., product description) and numeric features. Use text_features parameter. Show how to preprocess text (lowercase, remove punctuation) before feeding it to CatBoost."

Example Usage:

from catboost import CatBoostClassifier, Pool
import pandas as pd
import re

# Sample data
df = pd.DataFrame({
    'price': [100, 200, 150, 300],
    'description': ['Great product!', 'Not bad.', 'Excellent quality.', 'Poor item.'],
    'label': [1, 0, 1, 0]
})

# Simple text preprocessing
def clean_text(text):
    text = text.lower()
    text = re.sub(r'[^\w\s]', '', text)
    return text

df['description'] = df['description'].apply(clean_text)

X = df[['price', 'description']]
y = df['label']

model = CatBoostClassifier(
    iterations=50,
    text_features=['description'],
    verbose=0,
    random_seed=42
)
model.fit(X, y)
print('Trained with text features successfully.')

Explanation: CatBoost converts text into numerical features using its own tokenization and embedding techniques. This is simpler than using TF-IDF or word embeddings separately.

Conclusion

These 12 prompts cover the essential steps in a machine learning workflow using Scikit-learn, XGBoost, and CatBoost. From data preprocessing and scaling to hyperparameter tuning and model persistence, each prompt is designed to be immediately actionable. Start by copying the code examples, then adapt them to your own datasets. As you gain confidence, explore more advanced topics like custom loss functions or text feature handling. The key is to build reusable pipelines that save time and reduce errors. For further learning, refer to the official documentation: Scikit-learn (https://scikit-learn.org/stable/), XGBoost (https://xgboost.readthedocs.io/), and CatBoost (https://catboost.ai/en/docs/). Happy modeling!