Linear Algebra for Machine Learning: Complete Mathematical Foundation

Module: AI/ML Premium | Difficulty: PhD Level | Prerequisites: Calculus, Basic Proof Techniques

1. Vector Spaces and Linear Maps

1.1 Definition of Vector Spaces

1.2 Basis and Dimension

1.3 Inner Product Spaces

2. Eigenvalues and Eigenvectors

2.1 Fundamental Theorem

2.2 Properties of Eigenvalues

3. Singular Value Decomposition (SVD)

3.1 The SVD Theorem

3.2 Eckart-Young-Mirsky Theorem

4. Matrix Decompositions for ML

4.1 LU Decomposition

Complexity: for decomposition, for solving .

4.2 Cholesky Decomposition

4.3 QR Decomposition

Applications: Least squares, eigenvalue algorithms (QR iteration).

5. Positive Definite Matrices

5.1 Characterizations

6. Matrix Calculus for Optimization

6.1 Derivatives of Matrix Functions

7. Implementations in Python

7.1 SVD Implementation

import numpy as np
from scipy.linalg import svd, diagsvd

def compute_svd_analysis(A):
    """Complete SVD analysis with reconstruction error."""
    U, sigma, Vt = svd(A, full_matrices=False)
    
    # Reconstruction
    k_values = range(1, min(A.shape) + 1)
    errors = []
    
    for k in k_values:
        A_k = U[:, :k] @ np.diag(sigma[:k]) @ Vt[:k, :]
        error = np.linalg.norm(A - A_k, 'fro')
        errors.append(error)
    
    return {
        'U': U, 'sigma': sigma, 'Vt': Vt,
        'errors': errors,
        'condition_number': sigma[0] / sigma[-1],
        'effective_rank': np.sum(sigma > sigma[0] * 1e-10)
    }

# Example: Image compression
def compress_image_svd(image_matrix, k):
    """Compress image using truncated SVD."""
    U, sigma, Vt = svd(image_matrix, full_matrices=False)
    return U[:, :k] @ np.diag(sigma[:k]) @ Vt[:k, :]

7.2 Eigendecomposition

import numpy as np
from scipy.linalg import eigh

def spectral_analysis(A):
    """Spectral decomposition with analysis."""
    eigenvalues, eigenvectors = eigh(A)
    
    # Verify orthogonality
    orthogonality_error = np.linalg.norm(
        eigenvectors.T @ eigenvectors - np.eye(len(eigenvalues))
    )
    
    # Reconstruction
    Lambda = np.diag(eigenvalues)
    reconstructed = eigenvectors @ Lambda @ eigenvectors.T
    recon_error = np.linalg.norm(A - reconstructed)
    
    return {
        'eigenvalues': eigenvalues,
        'eigenvectors': eigenvectors,
        'spectral_radius': np.max(np.abs(eigenvalues)),
        'trace': np.sum(eigenvalues),
        'determinant': np.prod(eigenvalues),
        'condition_number': np.max(np.abs(eigenvalues)) / np.min(np.abs(eigenvalues)),
        'orthogonality_error': orthogonality_error,
        'reconstruction_error': recon_error
    }

8. Advanced Topics

8.1 Randomized SVD

8.2 Streaming SVD

Summary

Research Frontier: Randomized linear algebra enables scalable ML algorithms with provable guarantees, achieving near-optimal accuracy in sublinear time for many applications.