Model Selection & Hyperparameter Tuning

Choosing the right model and tuning it properly is crucial for ML success.

Algorithm Selection

Quick Guide:

Small dataset (<1K samples):
├─ SVM with RBF kernel
├─ KNN
├─ Naive Bayes
└─ Random Forest

Medium dataset (1K-100K):
├─ XGBoost / LightGBM
├─ Random Forest
├─ Neural Networks (simple)
└─ SVM with linear kernel

Large dataset (>100K):
├─ XGBoost / LightGBM
├─ Neural Networks
├─ Linear models
└─ SGDClassifier

High dimensional (features > samples):
├─ Linear models (L1/L2)
├─ SVM
└─ Naive Bayes

Interpretability needed:
├─ Decision Trees
├─ Linear/Logistic Regression
└─ Rule-based models

Hyperparameter Tuning

Grid Search:
├─ Try EVERY combination
├─ Guaranteed to find best in grid
├─ Exponentially expensive
└─ Use for small parameter spaces

Random Search:
├─ Random combinations
├─ Often finds good results faster
├─ Better use of budget
└─ Default choice for most cases

Bayesian Optimization:
├─ Uses past results to guide search
├─ Most efficient
├─ Best for expensive models
└─ Use library: Optuna

from sklearn.model_selection import GridSearchCV, RandomizedSearchCV
from sklearn.ensemble import RandomForestClassifier

# Grid Search
param_grid = {
    'n_estimators': [50, 100, 200],
    'max_depth': [5, 10, 20, None],
    'min_samples_split': [2, 5, 10]
}

grid = GridSearchCV(RandomForestClassifier(), param_grid, cv=5, scoring='accuracy')
grid.fit(X_train, y_train)
print(f"Best: {grid.best_params_}")

# Random Search (faster)
random = RandomizedSearchCV(RandomForestClassifier(), param_grid, n_iter=20, cv=5)
random.fit(X_train, y_train)

Optuna (Bayesian Optimization)

import optuna

def objective(trial):
    params = {
        'n_estimators': trial.suggest_int('n_estimators', 50, 300),
        'max_depth': trial.suggest_int('max_depth', 3, 20),
        'learning_rate': trial.suggest_float('learning_rate', 0.01, 0.3, log=True)
    }
    model = xgb.XGBClassifier(**params)
    return cross_val_score(model, X, y, cv=5).mean()

study = optuna.create_study(direction='maximize')
study.optimize(objective, n_trials=100)
print(f"Best params: {study.best_params}")

Key Takeaways

1. Start with simple models as baselines
2. Random search is usually better than grid search
3. Bayesian optimization (Optuna) is most efficient
4. Always use cross-validation for evaluation
5. XGBoost/LightGBM are often the best tabular models
6. Scale data for SVM, KNN, Neural Networks
7. Feature engineering matters more than model choice
8. Ensemble multiple models for best performance