Model Compression

Deep learning models are often over-parameterized for deployment. Model compression reduces model size and computation while maintaining accuracy, enabling deployment on edge devices and reducing inference costs.

Unstructured Pruning

DfUnstructured Pruning

Set individual weights to zero based on magnitude:

w_{ij} = \begin{cases} 0 & \text{if } |w_{ij}| < \theta \\ w_{ij} & \text{otherwise} \end{cases}

Creates sparse weight matrices that require specialized hardware/SparseTensor support for speedup.

Magnitude Pruning Threshold

w_{ij} = w_{ij} \cdot \mathbb{1}(|w_{ij}| \geq \theta)

Here,

$w_{ij}$ =Weight to prune
$\theta$ =Threshold (percentile of absolute weights)
$\mathbb{1}$ =Indicator function

Structured Pruning

DfStructured Pruning

Remove entire structures (filters, channels, heads, layers) rather than individual weights:

Filter pruning: Remove entire convolutional filters
Channel pruning: Remove output channels (equivalent to filter pruning)
Head pruning: Remove attention heads in Transformers
Layer pruning: Remove entire layers

Structured pruning produces smaller dense matrices that achieve actual speedup without specialized hardware.

Filter Importance (L1-norm)

I_f = \sum_{c,h,w} |W_{f,c,h,w}|

Here,

$I_f$ =Importance score for filter f
$W_{f,c,h,w}$ =Weight of filter f at position (c,h,w)

Quantization

DfQuantization

Reduce precision of weights and activations from FP32 to lower-bit representations:

FP16: 16-bit floating point (half precision) — 2x memory reduction
INT8: 8-bit integer — 4x memory reduction, 2-3x speedup
INT4: 4-bit integer — 8x memory reduction
Binary: 1-bit — extreme compression

Quantization-aware training (QAT) simulates quantization during training for better accuracy.

Linear Quantization

x_q = \text{round}\left(\frac{x}{\text{scale}}\right) + \text{zero\_point}

Here,

$x_q$ =Quantized value
$x$ =Original floating-point value
$scale$ =Scaling factor
$zero\_point$ =Zero-point offset

Scale and Zero Point

\text{scale} = \frac{x_{\max} - x_{\max}}{2^n - 1}, \quad \text{zero\_point} = \text{round}\left(-\frac{x_{\min}}{\text{scale}}\right)

Here,

$x_{\min}, x_{\max}$ =Range of floating-point values
$n$ =Number of bits (e.g., 8 for INT8)

Knowledge Distillation

\mathcal{L} = \alpha \cdot \mathcal{L}_{\text{CE}}(y, p_s) + (1 - \alpha) \cdot T^2 \cdot D_{\text{KL}}(p_t^{\text{soft}} \| p_s^{\text{soft}})

DfKnowledge Distillation

Distill knowledge from a large teacher model $T$ to a smaller student model $S$ :

Soft targets: Teacher's softened outputs with temperature $T$
Hard targets: Ground truth labels
Combined loss: Weighted sum of soft and hard losses

The soft targets contain "dark knowledge" — relationships between classes that hard labels miss.

Softened Probabilities

p_i^{\text{soft}} = \frac{\exp(z_i / T)}{\sum_j \exp(z_j / T)}

Here,

$z_i$ =Logit for class i
$T$ =Temperature (higher = softer distribution)
$p_i^{\text{soft}}$ =Softened probability

Model Compression — Pruning, Quantization, Distillation

Model Compression

Unstructured Pruning

DfUnstructured Pruning

Magnitude Pruning Threshold

Structured Pruning

DfStructured Pruning

Filter Importance (L1-norm)

Quantization

DfQuantization

Linear Quantization

Scale and Zero Point

Knowledge Distillation

DfKnowledge Distillation

Softened Probabilities

Need Expert Deep Learning Help?