Autoencoders — Complete Guide

Autoencoders learn compressed representations by encoding input to a bottleneck and decoding back.

Architecture

Input → Encoder → Bottleneck → Decoder → Output

Encoder: Compresses input to latent representation
Bottleneck: Compressed representation (fewer dimensions)
Decoder: Reconstructs input from bottleneck

Loss = ||Input - Output||² (reconstruction loss)

Types

Vanilla AE:
├─ Standard architecture
├─ Learns compressed representation
└─ Use for: Dimensionality reduction

Variational AE (VAE):
├─ Learns probability distribution in latent space
├─ Can generate new data by sampling
└─ Use for: Generation, anomaly detection

Denoising AE:
├─ Input is corrupted with noise
├─ Output is clean version
├─ Learns robust features
└─ Use for: Denoising, feature learning

Convolutional AE:
├─ Uses conv layers
├─ Better for images
└─ Use for: Image compression

VAE Implementation

class VAE(nn.Module):
    def __init__(self, input_dim, latent_dim):
        super().__init__()
        self.encoder = nn.Sequential(
            nn.Linear(input_dim, 256), nn.ReLU(),
            nn.Linear(256, 128), nn.ReLU()
        )
        self.mu = nn.Linear(128, latent_dim)
        self.logvar = nn.Linear(128, latent_dim)
        self.decoder = nn.Sequential(
            nn.Linear(latent_dim, 128), nn.ReLU(),
            nn.Linear(128, 256), nn.ReLU(),
            nn.Linear(256, input_dim), nn.Sigmoid()
        )

    def reparameterize(self, mu, logvar):
        std = torch.exp(0.5 * logvar)
        eps = torch.randn_like(std)
        return mu + eps * std

    def forward(self, x):
        h = self.encoder(x)
        mu, logvar = self.mu(h), self.logvar(h)
        z = self.reparameterize(mu, logvar)
        return self.decoder(z), mu, logvar

Key Takeaways

1. Autoencoders learn compressed representations
2. VAE enables generation by sampling latent space
3. Denoising AE learns robust features
4. Use for anomaly detection (high reconstruction error = anomaly)
5. Latent space enables interpolation between data points
6. Variational loss balances reconstruction and latent space regularization
7. Autoencoders are the foundation of diffusion models
8. Convolutional AE for image compression