Bayesian Optimization: Sequential Model-Based Optimization

Module: Machine Learning | Difficulty: Advanced

Optimization Problem

where is a black-box function.

Gaussian Process Surrogate

Expected Improvement (EI)

where

Upper Confidence Bound (UCB)

Knowledge Gradient

import numpy as np
from scipy.optimize import minimize as sp_minimize

class BayesianOptimizer:
    def __init__(self, bounds, n_initial=5):
        self.bounds = np.array(bounds)
        self.n_initial = n_initial
        self.X_observed = []
        self.y_observed = []
    def _expected_improvement(self, X, y_best, gp):
        mu, var = gp.predict(X)
        sigma = np.sqrt(var + 1e-9)
        z = (mu - y_best) / sigma
        ei = (mu - y_best) * norm.cdf(z) + sigma * norm.pdf(z)
        return ei
    def suggest(self, gp):
        if len(self.X_observed) < self.n_initial:
            return np.random.uniform(self.bounds[:,0], self.bounds[:,1])
        y_best = max(self.y_observed)
        result = sp_minimize(lambda x: -self._expected_improvement(x.reshape(1,-1), y_best, gp),
                           x0=np.random.uniform(self.bounds[:,0], self.bounds[:,1]),
                           bounds=self.bounds)
        return result.x

Research Insight: Bayesian optimization is most sample-efficient when the function is expensive to evaluate (e.g., neural network hyperparameters). The EI acquisition function is optimal under the assumption that the GP model is correct.