Hyperparameter Tuning

Introduction

Optimize model hyperparameters using grid search and random search.

Grid Search

from sklearn.model_selection import GridSearchCV

param_grid = {
    "n_estimators": [50, 100, 200],
    "max_depth": [3, 5, 10, None],
    "min_samples_split": [2, 5, 10]
}

grid_search = GridSearchCV(
    RandomForestClassifier(),
    param_grid,
    cv=5,
    scoring="accuracy",
    n_jobs=-1
)
grid_search.fit(X_train, y_train)

print(grid_search.best_params_)
print(grid_search.best_score_)

Random Search

from sklearn.model_selection import RandomizedSearchCV
from scipy.stats import randint, uniform

param_dist = {
    "n_estimators": randint(50, 200),
    "max_depth": [3, 5, 10, None],
    "learning_rate": uniform(0.01, 0.3)
}

random_search = RandomizedSearchCV(
    GradientBoostingClassifier(),
    param_dist,
    n_iter=50,
    cv=5,
    random_state=42
)
random_search.fit(X_train, y_train)

Practice Problems

1. Grid search for best parameters
2. Random search for large parameter space
3. Combine with cross-validation
4. Visualize parameter effects
5. Save and load best model