Tensor Calculus
Core Definitions
Key Formulas
Important Theorems
Worked Examples
Practice Problems
Common Mistakes
| Mistake | Correct Approach |
|---|---|
| Confusing (dot product) with (matrix multiply) | Check which indices are repeated vs. free |
Forgetting that einsum 'ij,kl->ijkl' is outer product, not contraction | Only repeated indices in the same term are summed |
| Assuming (matrix multiply is not commutative) | Order matters: in general |
| Using same index name for different dimensions | Each dummy index must appear exactly twice; use distinct names |
| Ignoring index alignment in batch operations | Batch dimensions must match or be broadcastable |
| Confusing covariant (lower) and contravariant (upper) indices | Upper indices transform with Jacobian, lower with inverse Jacobian |
| Forgetting that einsum does not broadcast | All dimensions must match or appear in the output |
Connections to Machine Learning
Exam/Interview Questions
Q1: Write the einsum notation for computing the matrix where and .
Answer: 'ik,jk->ij'. The summation is over (the shared dimension), and the free indices and give the row and column of .
Q2: Explain the difference between a covariant and contravariant tensor.
Answer: A covariant tensor (lower indices ) transforms with the Jacobian of the coordinate change: . A contravariant tensor (upper indices ) transforms with the inverse Jacobian: . Intuitively, covariant components shrink when the coordinate system expands, while contravariant components expand.
Q3: What is the tensor product of two vectors, and how does it relate to the outer product?
Answer: The tensor product produces a rank-2 tensor with components . This is identical to the outer product in linear algebra. The tensor product generalizes to higher-order tensors: the product of a rank-2 and rank-1 tensor gives a rank-3 tensor.
Q4: Why is the Frobenius norm of a matrix equal to ?
Answer: , so . Taking the square root gives the Frobenius norm. This identity is useful for computing norms without explicitly forming the full matrix.
Q5: In transformer attention, the operation is . Explain what tensor operations are involved.
Answer: is a batched matrix multiplication with einsum 'btd,bsd->bts', producing attention scores. Division by is elementwise scaling. Softmax is applied over the (source) dimension. The final multiplication with is 'bts,bsd->btd', a weighted sum over source positions. All operations are tensor contractions or elementwise.
Q6: What is the rank of a tensor, and why does it matter for model compression?
Answer: The tensor rank is the minimum number of rank-1 tensors needed to express a tensor as a sum: . A low-rank weight tensor can be decomposed into smaller factors, reducing the number of parameters from to . This is the basis of tensor decomposition methods for neural network compression.
Q7: How does backpropagation use tensor contractions?
Answer: Backpropagation applies the chain rule through each layer. For , the gradient is an outer product (tensor contraction with the input). For multi-dimensional tensors (e.g., convolutions), the gradient is computed via transposed convolution operations, which are also tensor contractions. Einsum notation makes these operations explicit and ensures correct index alignment.
Quick Reference
| Operation | Einstein Notation | Einsum String | Result Shape |
|---|---|---|---|
| Matrix Multiply | 'ik,kj->ij' | ||
| Batch MatMul | 'bik,bkj->bij' | ||
| Dot Product | 'i,i->' | scalar | |
| Outer Product | 'i,j->ij' | ||
| Trace | 'ii->' | scalar | |
| Frobenius Norm | 'ij,ij->' | scalar | |
| Transpose | 'ij->ji' | ||
| Diagonal | 'ii->i' | ||
| Sum | 'ij->' | scalar | |
| Elementwise Multiply | 'ij,ij->ij' | ||
| Matrix Norm | 'ij,kl->' | scalar | |
| Batch Trace | 'bii->b' |
Cross-References
- 097-advanced-differential-geometry β Metric tensors on manifolds extend the concept of inner products to curved spaces; Christoffel symbols are tensor expressions
- 098-advanced-functional-analysis β Tensor products of vector spaces generalize to infinite-dimensional Hilbert spaces; operator tensors in quantum mechanics
- 099-advanced-measure-theory β Integration of tensor fields requires measure-theoretic foundations; tensor-valued measures
- Linear Algebra (earlier modules) β Matrix operations, eigenvalues, and SVD are foundational to tensor decompositions
- Neural Networks: Backpropagation computes gradients as tensor contractions through the computational graph