What Is Deep Learning — Foundations & The Deep Learning Revolution

Deep learning is a subset of machine learning that uses artificial neural networks with multiple layers to learn hierarchical representations of data. It has transformed industries from healthcare to autonomous driving.

See our Machine Learning tutorial for a comprehensive introduction to classical ML methods.

What Is Deep Learning?

History: From Perceptrons to Deep Learning

The Perceptron Era (1958)

The AI Winter (1970s–1980s)

Minsky and Papert (1969) proved that single-layer perceptrons cannot solve the XOR problem, leading to a decades-long decline in neural network research. The field entered an "AI winter" as funding dried up.

Backpropagation Revival (1986)

Rumelhart, Hinton, and Williams popularized backpropagation for training multi-layer networks. This allowed networks to learn nonlinear decision boundaries, but training remained slow.

The Modern Deep Learning Revolution (2012–Present)

Three factors converged to enable the deep learning revolution:

Deep Learning vs. Traditional ML

Deep Learning Architecture Taxonomy

A Simple Deep Network in PyTorch

import torch
import torch.nn as nn
import torch.optim as optim
from sklearn.datasets import make_moons
from sklearn.model_selection import train_test_split

# Generate 2D classification data
X, y = make_moons(n_samples=1000, noise=0.2, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)

X_train = torch.FloatTensor(X_train)
y_train = torch.LongTensor(y_train)
X_test = torch.FloatTensor(X_test)
y_test = torch.LongTensor(y_test)

# Define a deep network
class DeepMLP(nn.Module):
    def __init__(self):
        super().__init__()
        self.network = nn.Sequential(
            nn.Linear(2, 64),
            nn.ReLU(),
            nn.Linear(64, 128),
            nn.ReLU(),
            nn.Linear(128, 64),
            nn.ReLU(),
            nn.Linear(64, 2)
        )

    def forward(self, x):
        return self.network(x)

# Training loop
model = DeepMLP()
criterion = nn.CrossEntropyLoss()
optimizer = optim.Adam(model.parameters(), lr=0.001)

for epoch in range(200):
    # Forward pass
    outputs = model(X_train)
    loss = criterion(outputs, y_train)

    # Backward pass
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

    if (epoch + 1) % 50 == 0:
        correct = (outputs.argmax(1) == y_train).float().mean()
        print(f"Epoch {epoch+1}: Loss={loss.item():.4f}, Acc={correct:.4f}")

# Evaluate on test set
with torch.no_grad():
    test_outputs = model(X_test)
    test_acc = (test_outputs.argmax(1) == y_test).float().mean()
    print(f"\nTest Accuracy: {test_acc:.4f}")

Deep Learning Frameworks

The Deep Learning Pipeline

Summary

Practice Exercises

1. Conceptual: Explain why the XOR problem cannot be solved by a single-layer perceptron but can be solved by a two-layer network. What does this tell us about the need for depth?

2. Coding: Modify the DeepMLP example above to add a 5th hidden layer with 256 neurons. Compare training speed and final accuracy with the 4-layer version.

3. Research: Look up the ImageNet Large Scale Visual Recognition Challenge (ILSVRC) results from 2012 to 2023. How has top-5 error rate changed? What drove the improvements?

4. Application: Download the CIFAR-10 dataset using torchvision. Build a CNN classifier and compare its performance with a simple MLP on the same data. What does this tell you about the inductive biases of CNNs?

5. Critical Thinking: The universal approximation theorem says a single hidden layer can approximate any function. Why don't we just use very wide single-layer networks instead of deep ones? Discuss computational and statistical reasons.