Linear Transformations
What is a Linear Transformation?
Matrix Representation
Every linear transformation can be written as matrix multiplication.
How to find A: The columns of are where are the standard basis vectors.
Kernel and Image
Common Transformations
Rotation
Scaling
Reflection
Shear
Projection
Summary Table:
| Transformation | Matrix | Determinant | Effect |
|---|---|---|---|
| Rotation (θ) | 1 | Rotates, preserves lengths | |
| Scaling () | Stretches/shrinks | ||
| Reflection | -1 | Flips, preserves lengths | |
| Shear | 1 | Slants | |
| Projection | 0 | Collapses dimension | |
| Zero | 0 | Maps everything to origin |
Composition of Transformations
The composition of two linear transformations corresponds to matrix multiplication.
Change of Basis
Eigenvalues and Transformations
Eigenvalues reveal the geometric behavior of a transformation.
Affine vs Linear Transformations
Python Implementation
import numpy as np
# Define a linear transformation matrix
A = np.array([[2, 1], [1, -1]])
# Apply transformation
x = np.array([1, 2])
Tx = A @ x # [4, -1]
# Rotation
theta = np.pi / 4 # 45 degrees
R = np.array([[np.cos(theta), -np.sin(theta)],
[np.sin(theta), np.cos(theta)]])
# Scale
S = np.array([[2, 0], [0, 0.5]])
# Composition: rotate then scale
M = S @ R
# Kernel (null space)
from scipy.linalg import null_space
kernel = null_space(A)
print(f"Kernel: {kernel.flatten()}")
# Image (column space)
from scipy.linalg import column_space
# Or simply: rank determines dimension
rank = np.linalg.matrix_rank(A)
print(f"Rank: {rank}, Image dimension: {rank}")
# Verify linearity: T(u + v) = T(u) + T(v)
u, v = np.array([1, 0]), np.array([0, 1])
lhs = A @ (u + v)
rhs = A @ u + A @ v
print(f"T(u+v) = T(u)+T(v): {np.allclose(lhs, rhs)}")
# Verify: T(cu) = cT(u)
c = 3
lhs = A @ (c * u)
rhs = c * (A @ u)
print(f"T(cu) = cT(u): {np.allclose(lhs, rhs)}")
Applications in AI/ML
| Application | Transformation | How It's Used |
|---|---|---|
| Neural Network Layer | Linear + bias, then activation | |
| Data Augmentation | Rotation, flip, crop | Increases training data diversity |
| Attention Mechanism | projection | Projects query-key similarity |
| PCA | Projection onto principal components | Dimensionality reduction |
| Word Embeddings | Linear map from vocabulary to semantic space | Maps one-hot to dense vectors |
| Computer Graphics | Rotation, scaling, projection | Rendering 3D scenes |
Common Mistakes
| Mistake | Why It's Wrong | Correct Approach |
|---|---|---|
| Assuming order doesn't matter | generally | Always specify transformation order |
| Forgetting non-linearity | Stack of linear transforms = one linear transform | Add activation between layers |
| Confusing kernel with image | Kernel = inputs->0, Image = all outputs | Use Rank-Nullity to relate them |
| Assuming determinant ≠ 0 means invertible over any field | True for ℝ but not always | Check rank for robustness |
| Using wrong basis for matrix | Matrix depends on basis | Use change-of-basis when switching |
Interview Questions
Q1: What's the kernel of a transformation and why does it matter? A: The kernel (null space) is the set of inputs mapping to zero. Its dimension tells you how much information is lost. In neural networks, a large kernel means many inputs produce the same output — bad for discrimination.
Q2: Why must neural networks have non-linearities? A: Without them, any number of layers collapses to a single linear transformation . Non-linearities (ReLU, sigmoid) break this collapse, allowing the network to learn complex decision boundaries.
Q3: What's the geometric meaning of a determinant of zero? A: The transformation collapses space to a lower dimension. The kernel is non-trivial (contains more than just ), and the transformation is not invertible.
Q4: How does data augmentation relate to transformations? A: Data augmentation applies random transformations (rotation, flip, crop) to training data. This makes the model invariant to those transformations, improving generalization.
Q5: What's the relationship between eigenvalues and transformation behavior? A: Eigenvalues tell you how the transformation scales along eigenvector directions. Positive eigenvalues stretch, negative eigenvalues reflect, zero eigenvalues collapse, and complex eigenvalues rotate.
Practice Problems
- Find the matrix representation of defined by .
- Find the kernel and image of .
- What transformation does represent?
- Is a linear transformation? Prove or disprove.
Quick Reference
| Transformation | Matrix | Properties |
|---|---|---|
| Rotation (θ) | Preserves lengths, det=1 | |
| Scaling () | Stretches, det= | |
| Reflection | Preserves lengths, det=-1 | |
| Shear | Preserves area, det=1 | |
| Projection | Collapses, det=0 |
| Property | Formula |
|---|---|
| Kernel | |
| Image | |
| Rank-Nullity | |
| Composition |