CNN Architecture Deep Dive

1. Receptive Field Theory

1.1 Receptive Field Computation

The receptive field of a neuron in a convolutional network is the region in the input space that influences that neuron's activation. For a network with convolutional layers, the receptive field grows according to:

where is the receptive field at layer , is the kernel size at layer , and is the stride at layer .

Effective Receptive Field. The effective receptive field (ERF) is the region that actually has significant influence, as opposed to the theoretical receptive field. For a network with receptive field radius , the effective radius typically scales as:

due to the Gaussian-like distribution of gradients from the centre (Luo et al., 2016).

1.2 Receptive Field Growth Strategies

1.3 Dilated (Atrous) Convolutions

Dilated convolutions insert zeros between kernel elements, expanding the receptive field without increasing parameters:

where is the dilation factor. A stack of convolutions with dilation yields:

2. Architectural Evolution

2.1 LeNet-5 (1998)

The foundational CNN architecture for digit recognition:

Input (32×32×1)
  → Conv(5×5, 6) → Tanh → AvgPool(2×2)
  → Conv(5×5, 16) → Tanh → AvgPool(2×2)
  → FC(120) → FC(84) → FC(10)

Parameter count: ~60K

2.2 AlexNet (2012)

The architecture that started the deep learning revolution:

• 5 convolutional layers + 3 fully-connected layers
• ReLU activation (first to use ReLU in a CNN)
• Dropout regularisation
• Data augmentation
• Parameter count: ~60M

2.3 VGGNet (2014)

Key insight: use small (3×3) convolutions throughout:

A stack of three 3×3 convolutions has the same receptive field as a 7×7 convolution but with:

• Fewer parameters: vs (45% reduction)
• More non-linearities: 3 vs 1
• Deeper representation

VGG-16 parameter count: ~138M

2.4 ResNet (2015)

The skip connection innovation that enabled training of very deep networks:

Gradient flow with skip connections:

The identity shortcut ensures that gradients can flow directly through the network, mitigating the vanishing gradient problem.

ResNet Variants:

2.5 DenseNet (2017)

Dense connections: each layer receives feature maps from all preceding layers:

where denotes concatenation.

Advantages:

• Strong gradient flow
• Feature reuse
• Fewer parameters per layer (with growth rate )

The number of parameters per layer is where is the number of input channels and is the growth rate.

2.6 EfficientNet (2019)

Compound Scaling: Scale depth, width, and resolution simultaneously:

subject to , where is the compound coefficient.

Base Architecture (EfficientNet-B0): MobileNetV3-based with squeeze-and-excitation.

2.7 ConvNeXt (2022)

A modernised ConvNet competing with Vision Transformers:

Design principles:

1. Inverted bottleneck (expand channels, then compress)
2. Large kernel size (7×7)
3. Fewer activation functions (GELU instead of ReLU)
4. Layer normalization instead of batch normalization

where DWConv is depthwise convolution.

3. Depthwise Separable Convolutions

3.1 Mathematical Formulation

A standard convolution with kernel applied to input has computational cost:

Depthwise separable convolution factorises this into:

1. Depthwise convolution: Apply independent filters

1. Pointwise convolution: convolution mixing channels

Total cost:

Compression ratio:

For , this gives a 8-9× reduction in computation.

3.2 MobileNet Architecture

MobileNet uses depthwise separable convolutions with width multiplier and resolution multiplier :

4. Feature Hierarchies and Visualisation

4.1 Hierarchical Feature Learning

CNNs learn hierarchical representations where:

• Early layers: Low-level features (edges, textures)
• Middle layers: Mid-level features (patterns, parts)
• Deep layers: High-level features (objects, scenes)

Class Activation Mapping (CAM): Visualise which regions contribute to a class prediction:

where is the -th activation map and is the weight for class .

4.2 Gradient-based Visualisation

Saliency maps: Compute gradient of output w.r.t. input:

Grad-CAM: Use gradients flowing into the last convolutional layer:

4.3 Feature Map Visualisation

import torch
import torch.nn as nn
import torchvision.models as models
import torchvision.transforms as transforms
from PIL import Image
import matplotlib.pyplot as plt

class FeatureVisualiser:
    """Visualise intermediate CNN feature maps."""
    
    def __init__(self, model, layer_names):
        self.model = model
        self.layer_names = layer_names
        self.activations = {}
        self.hooks = []
        
        for name in layer_names:
            layer = dict(model.named_modules())[name]
            hook = layer.register_forward_hook(
                self._get_hook(name)
            )
            self.hooks.append(hook)
    
    def _get_hook(self, name):
        def hook(module, input, output):
            self.activations[name] = output.detach()
        return hook
    
    def visualise(self, image_tensor, n_features=8):
        """Visualise feature maps for a given input."""
        self.model.eval()
        with torch.no_grad():
            self.model(image_tensor.unsqueeze(0))
        
        fig, axes = plt.subplots(
            len(self.layer_names), n_features,
            figsize=(n_features * 2, len(self.layer_names) * 2)
        )
        
        for i, name in enumerate(self.layer_names):
            acts = self.activations[name][0]  # first batch element
            
            for j in range(n_features):
                if j < acts.shape[0]:
                    ax = axes[i, j] if len(self.layer_names) > 1 else axes[j]
                    ax.imshow(acts[j].cpu().numpy(), cmap='viridis')
                    ax.axis('off')
                    if j == 0:
                        ax.set_ylabel(name, fontsize=10)
        
        plt.tight_layout()
        return fig

# Example: Visualise VGG-16 features
vgg = models.vgg16(pretrained=True)
layer_names = ['features.0', 'features.5', 'features.10', 'features.17']

# Load and preprocess image
transform = transforms.Compose([
    transforms.Resize((224, 224)),
    transforms.ToTensor(),
    transforms.Normalize(mean=[0.485, 0.456, 0.406],
                        std=[0.229, 0.224, 0.225])
])

image = Image.open('example.jpg')
image_tensor = transform(image)

# Visualise
visualiser = FeatureVisualiser(vgg, layer_names)
fig = visualiser.visualise(image_tensor, n_features=8)
fig.savefig('feature_maps.png', dpi=150, bbox_inches='tight')

5. Computational Complexity Analysis

5.1 FLOP Calculation

For a convolutional layer with input and output :

For a fully-connected layer:

5.2 Memory Footprint

Activation memory (training):

Parameter memory:

5.3 Throughput Estimation

For an A100 GPU (312 TFLOPS FP16, 2 TB/s bandwidth):

• Small model (ResNet-50): ~700 images/sec
• Large model (ViT-L): ~100 images/sec

6. Code Examples

6.1 Custom Residual Block with Receptive Field Tracking

import torch
import torch.nn as nn
from collections import OrderedDict

class ResidualBlock(nn.Module):
    """Residual block with receptive field tracking."""
    
    def __init__(self, in_channels, out_channels, stride=1, dilation=1):
        super().__init__()
        self.conv1 = nn.Conv2d(in_channels, out_channels, 3, stride, 
                               padding=dilation, dilation=dilation, bias=False)
        self.bn1 = nn.BatchNorm2d(out_channels)
        self.conv2 = nn.Conv2d(out_channels, out_channels, 3, 1,
                               padding=dilation, dilation=dilation, bias=False)
        self.bn2 = nn.BatchNorm2d(out_channels)
        self.relu = nn.ReLU(inplace=True)
        
        self.shortcut = nn.Sequential()
        if stride != 1 or in_channels != out_channels:
            self.shortcut = nn.Sequential(
                nn.Conv2d(in_channels, out_channels, 1, stride, bias=False),
                nn.BatchNorm2d(out_channels)
            )
        
        self.receptive_field = None
        self._compute_receptive_field(stride, dilation)
    
    def _compute_receptive_field(self, stride, dilation):
        """Compute receptive field for this block."""
        # For a single 3×3 conv with dilation d
        k_eff = 3 + (3 - 1) * (dilation - 1)  # effective kernel size
        self.receptive_field = {
            'kernel': k_eff,
            'stride': stride,
            'padding': dilation
        }
    
    def forward(self, x):
        residual = self.shortcut(x)
        out = self.relu(self.bn1(self.conv1(x)))
        out = self.bn2(self.conv2(out))
        out += residual
        return self.relu(out)


class ReceptiveFieldTracker:
    """Track receptive field through a network."""
    
    def __init__(self):
        self.rf = 1
        self.stride = 1
        self.history = []
    
    def update(self, kernel_size, stride, dilation=1):
        """Update receptive field after a conv layer."""
        k_eff = kernel_size + (kernel_size - 1) * (dilation - 1)
        self.rf += (k_eff - 1) * self.stride
        self.stride *= stride
        self.history.append({
            'rf': self.rf,
            'stride': self.stride,
            'kernel': kernel_size,
            'dilation': dilation
        })
        return self.rf
    
    def summary(self):
        """Print receptive field summary."""
        print(f"{'Layer':<20} {'RF':<10} {'Stride':<10}")
        print("-" * 40)
        for i, h in enumerate(self.history):
            print(f"Layer {i:<14} {h['rf']:<10} {h['stride']:<10}")
        print(f"\nTotal receptive field: {self.rf}")


# Example: Track RF through ResNet-50
tracker = ReceptiveFieldTracker()

# Initial conv: 7×7, stride 2, padding 3
tracker.update(7, stride=2, dilation=1)
# Max pool: 3×3, stride 2, padding 1
tracker.update(3, stride=2, dilation=1)

# Residual blocks
# Stage 1: 3 blocks, stride 1
for _ in range(3):
    tracker.update(3, stride=1, dilation=1)

# Stage 2: 4 blocks, first with stride 2
tracker.update(3, stride=2, dilation=1)
for _ in range(3):
    tracker.update(3, stride=1, dilation=1)

# Stage 3: 6 blocks, stride 1
for _ in range(6):
    tracker.update(3, stride=1, dilation=1)

# Stage 4: 3 blocks, stride 1
for _ in range(3):
    tracker.update(3, stride=1, dilation=1)

tracker.summary()

6.2 EfficientNet Compound Scaling

import torch
import torch.nn as nn
from math import ceil

class CompoundScaler:
    """
    EfficientNet compound scaling: scale depth, width, and resolution.
    
    Formulas:
        d = α^φ
        w = β^φ
        r = γ^φ
        
    Subject to: α · β² · γ² ≈ 2
    """
    
    def __init__(self, base_width=1.0, base_depth=1.0, base_resolution=224):
        self.base_width = base_width
        self.base_depth = base_depth
        self.base_resolution = base_resolution
    
    def compute_scaling_factors(self, phi, alpha=1.2, beta=1.1, gamma=1.15):
        """
        Compute scaling factors for given compound coefficient φ.
        
        Parameters
        ----------
        phi : int
            Compound coefficient (0-7 for B0-B7)
        alpha, beta, gamma : float
            Base scaling factors
        
        Returns
        -------
        width_factor, depth_factor, resolution_factor : float
        """
        width_factor = alpha ** phi
        depth_factor = beta ** phi
        resolution_factor = gamma ** phi
        
        # Verify constraint: α · β² · γ² ≈ 2
        constraint = alpha * (beta ** 2) * (gamma ** 2)
        
        return width_factor, depth_factor, resolution_factor, constraint
    
    def scale_model(self, base_model_config, phi):
        """
        Scale a base model configuration.
        
        Parameters
        ----------
        base_model_config : dict
            Base model configuration
        phi : int
            Compound coefficient
        
        Returns
        -------
        scaled_config : dict
            Scaled model configuration
        """
        w, d, r, constraint = self.compute_scaling_factors(phi)
        
        scaled_config = {
            'width': ceil(self.base_width * w),
            'depth': ceil(self.base_depth * d),
            'resolution': ceil(self.base_resolution * r / 32) * 32,  # round to 32
        }
        
        print(f"B{phi}: width={w:.3f}x, depth={d:.3f}x, resolution={r:.3f}x")
        print(f"  Constraint α·β²·γ² = {constraint:.3f} (should be ≈2)")
        print(f"  Scaled: {scaled_config}")
        
        return scaled_config


# Example: Generate configurations for B0-B7
scaler = CompoundScaler()

print("EfficientNet Compound Scaling")
print("=" * 50)

for phi in range(8):
    config = scaler.scale_model({}, phi)
    print()

6.3 Depthwise Separable Convolution

import torch
import torch.nn as nn

class DepthwiseSeparableConv(nn.Module):
    """
    Depthwise separable convolution.
    
    Decomposes standard convolution into:
    1. Depthwise: C_in independent k×k filters
    2. Pointwise: 1×1 convolution mixing channels
    
    FLOPs reduction: ~k²× reduction
    """
    
    def __init__(self, in_channels, out_channels, kernel_size=3, stride=1, 
                 padding=1, dilation=1):
        super().__init__()
        
        # Depthwise convolution
        self.depthwise = nn.Conv2d(
            in_channels, in_channels, kernel_size,
            stride=stride, padding=padding, dilation=dilation,
            groups=in_channels, bias=False
        )
        
        # Pointwise convolution
        self.pointwise = nn.Conv2d(
            in_channels, out_channels, 1, bias=False
        )
        
        self.bn1 = nn.BatchNorm2d(in_channels)
        self.bn2 = nn.BatchNorm2d(out_channels)
        self.relu = nn.ReLU(inplace=True)
        
        # Compute parameter counts
        self.dw_params = kernel_size * kernel_size * in_channels
        self.pw_params = in_channels * out_channels
        self.std_params = kernel_size * kernel_size * in_channels * out_channels
        
    def forward(self, x):
        x = self.relu(self.bn1(self.depthwise(x)))
        x = self.relu(self.bn2(self.pointwise(x)))
        return x
    
    def compression_ratio(self):
        """Compute compression ratio vs standard convolution."""
        return self.std_params / (self.dw_params + self.pw_params)


# Example: Compare standard vs depthwise separable
in_ch, out_ch, k = 64, 128, 3

# Standard convolution
std_conv = nn.Conv2d(in_ch, out_ch, k, padding=k//2)

# Depthwise separable
dw_sep_conv = DepthwiseSeparableConv(in_ch, out_ch, k)

# Compare
x = torch.randn(1, in_ch, 56, 56)

print("Standard Convolution:")
print(f"  Parameters: {sum(p.numel() for p in std_conv.parameters()):,}")
std_out = std_conv(x)
print(f"  Output shape: {std_out.shape}")

print("\nDepthwise Separable Convolution:")
print(f"  Parameters: {sum(p.numel() for p in dw_sep_conv.parameters()):,}")
dw_out = dw_sep_conv(x)
print(f"  Output shape: {dw_out.shape}")

print(f"\nCompression ratio: {dw_sep_conv.compression_ratio():.2f}x")

7. Summary

1. Receptive fields grow with network depth; dilated convolutions enable exponential growth.

2. Depth efficiency: Deep networks with skip connections can represent functions that require exponentially more parameters in shallow networks.

3. Modern architectures (EfficientNet, ConvNeXt) combine ideas from both CNNs and Transformers.

4. Depthwise separable convolutions reduce computation by ~ while maintaining accuracy.

5. Feature hierarchies in CNNs learn increasingly abstract representations from edges to objects.

References

• He, K., et al. (2016). Deep residual learning for image recognition. CVPR.
• Huang, G., et al. (2017). Densely connected convolutional networks. CVPR.
• Tan, M., & Le, Q. (2019). EfficientNet: Rethinking model scaling for CNNs. ICML.
• Liu, Z., et al. (2022). A ConvNet for the 2020s. CVPR.
• Luo, W., et al. (2016). Understanding the effective receptive field in deep convolutional neural networks. NeurIPS.
• Howard, A., et al. (2017). MobileNets: Efficient CNNs for mobile vision applications. arXiv.