Hyperparameter Tuning: Grid Search, Random Search and Optuna
ℹ️Module 8 — Tree-Based Models
This lesson covers systematic approaches to hyperparameter optimization, from exhaustive grid search to intelligent Bayesian methods and modern tools like Optuna.
1. Hyperparameters vs Parameters
Understanding the distinction between hyperparameters and learned parameters is fundamental to model optimization.
Parameters are learned from data during training — weights in a neural network, split thresholds in a decision tree. Hyperparameters control the learning process itself and must be set before training begins.
| Aspect | Parameters | Hyperparameters |
|---|---|---|
| Set when | During training | Before training |
| Learned from | Data | Manual / search |
| Examples | Weights, biases, split points | Learning rate, max depth, n_estimators |
| Optimized via | Gradient descent, EM | Grid search, random search, Bayesian |
Formal Definition
A machine learning model has parameters learned from training data via:
Hyperparameters govern the learning process:
The goal of hyperparameter tuning is to find:
where is the validation loss — we never use test data for this.
Why Tuning Matters
A well-tuned model can outperform a more complex model with default settings. The bias-variance tradeoff is directly controlled by hyperparameters:
- Too restrictive (e.g.,
max_depth=2): high bias, underfitting - Too flexible (e.g.,
max_depth=50): high variance, overfitting - Just right: optimal generalization
2. Grid Search — Exhaustive Exploration
Grid Search is the most straightforward approach: define a discrete set of values for each hyperparameter and evaluate every possible combination.
Algorithm
Python Implementation
The Curse of Dimensionality in Grid Search
With hyperparameters and values per parameter, grid search evaluates combinations:
| Hyperparameters | Values each | Combinations | Time (10s each) |
|---|---|---|---|
| 2 | 5 | 25 | 4 min |
| 3 | 5 | 125 | 21 min |
| 4 | 5 | 625 | 1.7 hours |
| 5 | 5 | 3,125 | 8.7 hours |
| 6 | 5 | 15,625 | 4.3 days |
Grid Search vs Random Search
The visual below demonstrates how random search covers the search space more efficiently. Each axis represents one hyperparameter, and colored dots represent evaluations.
Key Insight: Grid search wastes budget on unimportant hyperparameters. If max_depth matters more than subsample, grid search still evaluates all subsample values for every max_depth.
3. Random Search — Efficient Sampling
Random search samples hyperparameter combinations from specified distributions. Bergstra and Bengio (2012) showed random search is more efficient than grid search when only a few hyperparameters are truly important.
Why Random Search Wins
The key insight: if one hyperparameter (e.g., learning rate) dominates performance, grid search wastes evaluations per dimension. Random search explores the dominant dimension more effectively.
Log-Uniform Distributions for Learning Rate
Learning rates span orders of magnitude, so uniform sampling is suboptimal:
Comparison: Grid vs Random
| Criterion | Grid Search | Random Search |
|---|---|---|
| Coverage | Uniform grid | Stratified sampling |
| Curse of dimensionality | Exponential | Linear in n_iter |
| Important dimensions | Wasted budget | Better coverage |
| Reproducibility | Deterministic | Depends on seed |
| Parallelization | Difficult | Trivial |
| Budget efficiency | Low | High |
4. Bayesian Optimization — Intelligent Search
Bayesian optimization builds a surrogate model of the objective function and uses an acquisition function to decide where to sample next.
The Loop
Gaussian Process Surrogate
The surrogate model is typically a Gaussian Process (GP):
where is the mean function and is the kernel (covariance function). Given observations , the posterior predictive is:
Acquisition Functions
The acquisition function balances exploration and exploitation:
Expected Improvement (EI):
where is the best observed value, and are the standard normal CDF and PDF.
Upper Confidence Bound (UCB):
where controls exploration. Higher = more exploration.
Thompson Sampling: Sample and optimize the sample.
Exploration vs Exploitation
The acquisition function encodes a fundamental tradeoff:
- Exploration: sample where uncertainty is high (learning the landscape)
- Exploitation: sample where predicted value is good (refining the optimum)
EI naturally balances both: high increases exploitation, high increases exploration (only when is near ).
5. Optuna — State-of-the-Art Optimization
Optuna uses Tree-structured Parzen Estimator (TPE) and supports advanced features like pruning, conditional hyperparameters, and rich visualization.
TPE Algorithm
TPE models instead of — a key departure from Gaussian Process approaches:
- Split observations into "good" () and "bad" () using quantile
- Model good observations:
- Model bad observations:
- Maximize ratio:
TPE is non-parametric (uses kernel density estimation), scales better than GP-based methods, and naturally handles conditional hyperparameters.
Basic Usage
Conditional Hyperparameters
Optuna handles conditional spaces natively — parameters that only apply when another parameter takes a specific value:
Pruning — Early Termination of Bad Trials
Pruning terminates unpromising trials early, saving computational budget:
Pruning Strategies
| Pruner | Mechanism | Best For |
|---|---|---|
| MedianPruner | Prune if below median of previous trials | General purpose |
| SuccessiveHalvingPruner | Eliminate bottom fraction each round | Large search spaces |
| HyperbandPruner | Budget allocation with early stopping | Resource-constrained |
| PatientPruner | Wait N trials before pruning | Noisy objectives |
Optuna Visualization
6. Learning Rate Schedules
Learning rate scheduling reduces the learning rate during training, allowing fast convergence early and fine-grained updates late.
Common Schedules
where is the initial learning rate and is the current step.
| Schedule | Formula | Characteristics |
|---|---|---|
| Step Decay | Reduce by factor every steps | |
| Exponential Decay | Smooth continuous decay | |
| Cosine Annealing | Periodic warm restarts | |
| Linear Warmup | for | Avoid early instability |
| Polynomial Decay | Flexible power control |
Practical Implementation
7. Early Stopping
Early stopping halts training when validation performance stops improving, preventing overfitting and saving compute.
Mathematical Formulation
Let be the validation loss at epoch . Training stops when:
where is the patience parameter and is a tolerance threshold.
Implementation
Early Stopping for Gradient Boosting
Patience and Overfitting
8. Implementation in Python — Complete Pipeline
End-to-End Tuning Pipeline
Multi-Objective Optimization
Saving and Loading Studies
Key Takeaways
ℹ️Summary
Grid Search is simple but exponential — use only when search space is small (3-4 parameters).
Random Search is efficient for low-effective-dimension problems — always better than grid with equal budget.
Bayesian Optimization is sample-efficient — best when evaluations are expensive (deep learning, hyperparameter tuning of expensive models).
Optuna with TPE is the modern standard — handles conditional parameters, pruning, multi-objective, and scales to hundreds of trials.
Early stopping is cheap insurance — always use it for iterative learners like gradient boosting.
Learning rate schedules enable fast convergence with fine-tuned solutions — cosine annealing with warm restarts is often a strong default.
References
- Bergstra, J., and Bengio, Y. (2012). Random search for hyper-parameter optimization. JMLR, 13, 281-305.
- Snoek, J., Larochelle, H., and Adams, R. P. (2012). Practical Bayesian optimization of machine learning algorithms. NeurIPS.
- Akiba, T., et al. (2019). Optuna: A next-generation hyperparameter optimization framework. KDD.
- Li, L., et al. (2018). Hyperband: A novel bandit-based approach to hyperparameter optimization. JMLR, 18(185), 1-52.
- Smith, L. N. (2017). Cyclical learning rates for training neural networks. WACV.