Factor Analysis — Latent Variable Models

Factor analysis identifies latent (hidden) variables that explain correlations among observed variables. It reduces many correlated measurements into a smaller set of underlying factors.

Factor Loadings

Factor loadings represent the correlation between each observed variable and each factor. They form the loading matrix .

Factor Extraction Methods

Principal Component Method

The most common approach. Factors are extracted sequentially to maximize variance explained.

| Method | Key Idea | When to Use |

|--------|----------|-------------|

| Principal Components | Maximize total variance | Default; data reduction |

| Maximum Likelihood | Maximize likelihood under normality | Normal data; hypothesis testing |

| Principal Axis Factoring | Iteratively estimates communalities | When normality is questionable |

| Minimum Residual | Minimize off-diagonal residuals | Small samples |

Rotation

Rotation makes factors more interpretable by achieving simple structure — each variable loads highly on one factor and lowly on others.

Types of Rotation

| Type | Constraint | When to Use |

|------|-----------|-------------|

| Varimax (orthogonal) | Factors uncorrelated | When factors are independent |

| Promax (oblique) | Factors may correlate | When factors are expected to be related |

| Oblimin (oblique) | Factors may correlate | Flexible oblique rotation |

Number of Factors

Several criteria help determine how many factors to retain:

1. Kaiser's Rule: Retain factors with eigenvalues > 1

2. Scree Plot: Look for the "elbow" where eigenvalues level off

3. Parallel Analysis: Compare eigenvalues to those from random data

4. Velicer's MAP: Minimize average partial correlations

Assumptions

• Linearity: Relationships among variables are linear

• Multivariate normality: Data are approximately normally distributed

• Adequate sample size: Generally n > 100 (some say n > 5 variables per factor)

• No perfect multicollinearity: Variables are not perfectly correlated

KMO Test

The Kaiser-Meyer-Olkin measure assesses sampling adequacy.

| KMO Value | Interpretation |

|-----------|---------------|

| 0.9 - 1.0 | Marvelous |

| 0.8 - 0.9 | Meritorious |

| 0.7 - 0.8 | Middling |

| 0.6 - 0.7 | Mediocre |

| 0.5 - 0.6 | Miserable |

| < 0.5 | Unacceptable |

Python Implementation

import numpy as np

import pandas as pd

from factor_analyzer import FactorAnalyzer

from factor_analyzer.factor_analyzer import calculate_bartlett_sphericity, calculate_kmo



np.random.seed(42)



# Simulated data with 3 latent factors

n = 500

f1 = np.random.randn(n)

f2 = np.random.randn(n)

f3 = np.random.randn(n)



data = pd.DataFrame({

    'X1': 0.8*f1 + 0.1*np.random.randn(n),

    'X2': 0.7*f1 + 0.2*np.random.randn(n),

    'X3': 0.6*f1 + 0.3*np.random.randn(n),

    'X4': 0.8*f2 + 0.1*np.random.randn(n),

    'X5': 0.7*f2 + 0.2*np.random.randn(n),

    'X6': 0.6*f2 + 0.3*np.random.randn(n),

    'X7': 0.9*f3 + 0.1*np.random.randn(n),

    'X8': 0.7*f3 + 0.2*np.random.randn(n),

})



# Bartlett's test

chi_square, p_value = calculate_bartlett_sphericity(data)

print(f"Bartlett's test: chi2={chi_square:.2f}, p={p_value:.4e}")



# KMO

kmo_all, kmo_model = calculate_kmo(data)

print(f"KMO: {kmo_model:.3f}")



# Factor analysis with 3 factors, varimax rotation

fa = FactorAnalyzer(n_factors=3, rotation='varimax')

fa.fit(data)



# Loadings

loadings = pd.DataFrame(fa.loadings_,

    index=data.columns,

    columns=['Factor 1', 'Factor 2', 'Factor 3'])

print("\nFactor Loadings:")

print(loadings.round(3))



# Variance explained

variance = fa.get_factor_variance()

print(f"\nVariance explained: {variance[1].round(3)}")

print(f"Cumulative: {variance[2].round(3)}")

Worked Example

Key Takeaways

Related Topics

• See Principal Component Analysis for a related dimensionality reduction technique

• See Reliability Analysis for assessing measurement consistency

• See Structural Equation Modeling for confirmatory factor analysis