Lagrange Multipliers

In real-world ML pipelines, unconstrained optimization is the exception, not the rule. Every time you add a constraint — a budget on inference time, a fairness requirement across demographic groups, a maximum norm on weight vectors — you enter the domain of constrained optimization. Lagrange multipliers are the foundational tool that makes these problems tractable, and understanding them is essential for anyone building or analyzing ML systems.

What are Lagrange Multipliers

Geometric Interpretation

At the point of tangency:

is perpendicular to the level curve of
is perpendicular to the constraint curve
Since both gradients are perpendicular to the same tangent line, they must be parallel
Therefore for some scalar

If , increasing the constraint (relaxing it in the direction of ) increases . If , relaxing the constraint decreases . This makes the sensitivity of the optimal value to the constraint.

Method of Lagrange Multipliers

Single Constraint Examples

Multiple Constraints

KKT Conditions

Saddle Points

Python Implementation

import numpy as np
from scipy.optimize import minimize, minimize_scalar

# ============================================
# Example 1: Single Equality Constraint
# Minimize x^2 + y^2 subject to x + y = 1
# ============================================

def lagrange_single_constraint():
    f = lambda x: x[0]**2 + x[1]**2
    constraint = {'type': 'eq', 'fun': lambda x: x[0] + x[1] - 1}
    
    result = minimize(f, x0=[0.5, 0.5], constraints=constraint)
    print(f"Optimal point: x = {result.x[0]:.4f}, y = {result.x[1]:.4f}")
    print(f"Optimal value: {result.fun:.4f}")
    print(f"Multiplier (approx): lambda = {2 * result.x[0]:.4f}")
    # Analytical solution: x = y = 1/2, lambda = 1

lagrange_single_constraint()

# ============================================
# Example 2: Minimize distance to a plane
# Minimize x^2 + y^2 + z^2 subject to 2x + y - 2z = 4
# ============================================

def closest_point_on_plane():
    f = lambda x: x[0]**2 + x[1]**2 + x[2]**2
    constraint = {'type': 'eq', 'fun': lambda x: 2*x[0] + x[1] - 2*x[2] - 4}
    
    result = minimize(f, x0=[1, 1, 1], constraints=constraint)
    print(f"\nClosest point: ({result.x[0]:.4f}, {result.x[1]:.4f}, {result.x[2]:.4f})")
    print(f"Distance: {np.sqrt(result.fun):.4f}")
    # Analytical: (8/9, 4/9, -8/9), distance = 4*sqrt(5)/3

closest_point_on_plane()

# ============================================
# Example 3: Multiple constraints
# Minimize x^2 + y^2 + z^2 subject to x+y+z=1 and x-y=0
# ============================================

def multiple_constraints():
    f = lambda x: x[0]**2 + x[1]**2 + x[2]**2
    constraints = [
        {'type': 'eq', 'fun': lambda x: x[0] + x[1] + x[2] - 1},
        {'type': 'eq', 'fun': lambda x: x[0] - x[1]}
    ]
    
    result = minimize(f, x0=[1/3, 1/3, 1/3], constraints=constraints)
    print(f"\nOptimal point: ({result.x[0]:.4f}, {result.x[1]:.4f}, {result.x[2]:.4f})")
    print(f"Optimal value: {result.fun:.4f}")

multiple_constraints()

# ============================================
# Example 4: KKT with inequality constraints
# Minimize x^2 + y^2 subject to x + y >= 1
# ============================================

def kkt_inequality():
    f = lambda x: x[0]**2 + x[1]**2
    # Convert >= to <=: -(x + y - 1) <= 0
    constraint = {'type': 'ineq', 'fun': lambda x: x[0] + x[1] - 1}
    
    result = minimize(f, x0=[0, 0], constraints=constraint)
    print(f"\nKKT optimal point: ({result.x[0]:.4f}, {result.x[1]:.4f})")
    print(f"Optimal value: {result.fun:.4f}")
    # Solution: x = y = 0.5, value = 0.5

kkt_inequality()

Applications in AI/ML

Support Vector Machines (SVM)

Regularization as Constraint

Fairness Constraints

Portfolio Optimization

In finance-inspired ML, the Markowitz mean-variance portfolio optimizes return subject to risk constraints. Lagrange multipliers solve this, and the resulting efficient frontier maps out the tradeoff between risk and return — parameterized by .

Common Mistakes

Mistake	Why It's Wrong	Correct Approach
Forgetting the constraint equation	Solving alone gives equations for unknowns	Always include as an equation
Ignoring the sign of	vs indicates whether relaxing the constraint helps or hurts	Track the sign; it has physical meaning
Confusing with	Both conventions work, but mixing them leads to sign errors in	Pick one convention and be consistent
Assuming all critical points are extrema	Lagrange finds stationary points of , including saddle points	Verify with second-order conditions
Forgetting complementary slackness (KKT)	With inequality constraints, assuming all is wrong	Check: either or
Not checking constraint qualification	If at the candidate, the method may fail	Verify regularity (e.g., )
Using Lagrange for unconstrained problems	Lagrange multipliers are for constrained optimization	Use gradient-based methods for unconstrained problems
Ignoring boundary effects	The Lagrangian method finds interior critical points, not boundary solutions	Check boundary conditions separately if the domain is bounded

Interview Questions

Q1: What is the geometric interpretation of a Lagrange multiplier?

Q2: Why does the Lagrange multiplier method work? What is the intuition behind ?

Q3: How do Lagrange multipliers relate to the dual problem in SVM?

Q4: What is complementary slackness and why does it matter?

Q5: When might Lagrange multipliers fail or give misleading results?

Q6: Explain the relationship between Lagrange multipliers and sensitivity analysis.

Practice Problems

Problem 1: Box Optimization

Problem 2: Nearest Point on an Ellipsoid

Problem 3: Constrained Exponential

Problem 4: KKT with Two Inequality Constraints

Quick Reference

Cross-References

Optimization Fundamentals: Core optimization theory and unconstrained methods -> Optimization Fundamentals
Constrained Optimization: KKT conditions, penalty methods, and interior-point methods -> Constrained Optimization
Partial Derivatives: Gradients and partial derivatives needed for the Lagrangian -> Partial Derivatives
Multivariable Calculus: Gradients, Jacobians, and Hessians in higher dimensions -> Multivariable Calculus
Gradient Descent: Gradient-based optimization that complements Lagrange methods -> Gradient Descent
Linear Algebra Norms: Norms used in regularization constraints -> Linear Algebra Norms
Matrix Calculus: Derivatives of matrix-valued functions in SVM derivations -> Matrix Calculus
Probability Foundations: Probability constraints in ML fairness -> Probability Foundations

Lagrange Multipliers