ResNet, EfficientNet, and Modern Classification Architectures

Module: Computer Vision | Difficulty: Premium

Residual Learning

Instead of learning , learn the residual :

The identity shortcut ensures is optimal, making optimization easier.

Compound Scaling (EfficientNet)

subject to .

Label Smoothing

Reduces overconfidence and improves calibration.

CutMix Data Augmentation

| Model | Top-1 | Top-5 | Params | FLOPs | |-------|-------|-------|--------|-------| | ResNet-50 | 76.1% | 92.9% | 25M | 4.1G | | DenseNet-121 | 74.8% | 92.2% | 8M | 2.9G | | EfficientNet-B0 | 77.1% | 93.3% | 5.3M | 0.39G | | EfficientNet-B7 | 84.3% | 97.0% | 66M | 37G | | ViT-L/16 | 87.8% | 98.5% | 307M | 81G |

import torch
import torch.nn as nn

class ResidualBlock(nn.Module):
    def __init__(self, channels, downsample=False):
        super().__init__()
        stride = 2 if downsample else 1
        self.conv1 = nn.Conv2d(channels, channels, 3, stride, 1, bias=False)
        self.bn1 = nn.BatchNorm2d(channels)
        self.conv2 = nn.Conv2d(channels, channels, 3, 1, 1, bias=False)
        self.bn2 = nn.BatchNorm2d(channels)
        self.relu = nn.ReLU(inplace=True)
        self.shortcut = nn.Sequential()
        if downsample:
            self.shortcut = nn.Sequential(
                nn.Conv2d(channels, channels*2, 1, 2, bias=False),
                nn.BatchNorm2d(channels*2))

    def forward(self, x):
        out = self.relu(self.bn1(self.conv1(x)))
        out = self.bn2(self.conv2(out))
        return self.relu(out + self.shortcut(x))

def mixup_data(x, y, alpha=0.2):
    lam = torch.distributions.Beta(alpha, alpha).sample()
    batch_size = x.size(0)
    index = torch.randperm(batch_size)
    mixed = lam * x + (1 - lam) * x[index]
    return mixed, y, y[index], lam

def mixup_criterion(criterion, pred, y_a, y_b, lam):
    return lam * criterion(pred, y_a) + (1 - lam) * criterion(pred, y_b)

Research Insight: The scaling laws discovered by EfficientNet revealed that depth, width, and resolution are not independent axes but interact multiplicatively. This led to the realization that optimal architecture design is a multi-objective optimization problem. Recent work on foundation models (DINOv2, SAM) shows that pretraining on large diverse datasets yields representations that transfer broadly, potentially making task-specific architecture design less critical than data and compute scaling.