Interview Question

"Explain the mathematical foundation of PCA. How do you determine the number of components to keep? What are the limitations of PCA and when would you use alternatives like t-SNE or UMAP?"

Difficulty: Hard | Frequently asked at NVIDIA, Meta, Google

Theoretical Foundation

What is PCA?

Principal Component Analysis (PCA) is a dimensionality reduction technique that finds orthogonal directions (principal components) of maximum variance in the data.

Goal: Find a linear projection that preserves maximum variance.

Mathematical Formulation

Given centered data (columns have zero mean):

1. Compute covariance matrix:

2. Eigenvalue decomposition:

   where is the -th eigenvector and is the corresponding eigenvalue.

3. Sort by eigenvalues:

4. Projection: where contains the top eigenvectors.

Variance Explained

The proportion of variance explained by the -th component:

Cumulative variance explained:

Choosing the Number of Components

1. Variance Threshold

Keep components that explain at least of variance (typically 90-95%).

2. Scree Plot

Plot eigenvalues and look for an "elbow" where the rate of decrease slows.

3. Kaiser's Rule

Keep components with eigenvalues > 1 (for standardized data).

4. Cross-Validation

Use downstream task performance to choose the number of components.

PCA Algorithms

1. Full PCA (Eigenvalue Decomposition)

• Computes all eigenvectors
• Time complexity:
• Best for

2. Truncated SVD

• Computes only top components
• Time complexity:
• Best for large or sparse data

3. Randomized PCA

• Uses random projection + power iteration
• Time complexity:
• Best for very large datasets

Limitations of PCA

1. Linear Only: Cannot capture non-linear relationships
2. Variance-Based: May discard important features with low variance
3. Sensitive to Scaling: Features must be standardized
4. Interpretability: Components are linear combinations of original features
5. Outlier Sensitivity: Outliers can dominate principal components

Alternatives to PCA

1. t-SNE (t-Distributed Stochastic Neighbor Embedding)

• Preserves local structure (neighbors remain close)
• Non-linear dimensionality reduction
• Good for visualization (2-3D)
• Slow for large datasets

2. UMAP (Uniform Manifold Approximation and Projection)

• Preserves both local and global structure
• Faster than t-SNE
• Better preserves global geometry
• Can be used for general dimensionality reduction

3. Kernel PCA

• Uses kernel trick for non-linear PCA
• Maps to high-dimensional space
• Computationally expensive

4. Autoencoders

• Neural network-based dimensionality reduction
• Can learn complex non-linear mappings
• Requires more data and tuning

When to Use Each Method

Code Implementation

Explanation of Code

1. Basic PCA: Shows variance explained and component loadings.

2. Choosing Components: Demonstrates different methods for selecting the number of components.

3. Scree Plot: Visualizes variance explained and 2D projection.

4. Noise Reduction: Shows how PCA can denoise data by keeping only important components.

5. t-SNE Comparison: Compares PCA with t-SNE for visualization.

6. Kernel PCA: Demonstrates PCA on non-linear data.

7. ML Pipeline: Shows PCA as a preprocessing step for classification.

Real-World Applications

NVIDIA: Image Compression

NVIDIA uses PCA for:

• Feature Compression: Reducing CNN feature dimensions
• GPU Memory Optimization: Compressing activations
• Data Augmentation: PCA-based color augmentation

Meta: Dimensionality Reduction

Meta uses PCA for:

• Face Recognition: Eigenfaces approach
• Recommendation Systems: Reducing user/item embeddings
• Ad Targeting: Compressing user features

Common Follow-Up Questions

Q1: Why is scaling necessary before PCA?

PCA finds directions of maximum variance. If features have different scales, features with larger scales will dominate. Standardizing ensures all features contribute equally.

Q2: What is the difference between PCA and SVD?

SVD decomposes . PCA computes eigenvectors of . They're mathematically equivalent, but SVD is more numerically stable.

Q3: Can PCA be used for classification?

PCA is unsupervised and maximizes variance, not class separability. Linear Discriminant Analysis (LDA) is better for classification as it maximizes between-class variance.

Q4: How does PCA handle missing values?

Standard PCA cannot handle missing values. You must impute first or use robust PCA variants that handle missing data.

Company-Specific Tips

NVIDIA Interview Tips

• Discuss GPU-accelerated PCA implementations
• Be ready to explain truncated SVD for sparse data
• Mention randomized PCA for large datasets
• Talk about online PCA for streaming data

Meta Interview Tips

• Focus on scalability to billions of data points
• Discuss distributed PCA implementations
• Be prepared to explain incremental PCA
• Mention PCA for privacy (differential privacy)

Related Topics

• t-SNE and UMAP Deep Dive
• Feature Engineering Best Practices
• Linear Discriminant Analysis
• Autoencoders for Dimensionality Reduction