What Is Deep Learning - Foundations and The Deep Learning Revolution

Deep learning is a subset of machine learning that uses artificial neural networks with multiple layers to learn hierarchical representations of data. It has transformed industries from healthcare to autonomous driving.

What Is Deep Learning?

Writing one layer out in components makes the affine-plus-nonlinearity pattern explicit:

Training means finding the parameters that minimize an empirical risk over training samples:

and every optimizer in this course, from plain SGD to AdamW, is a rule for taking one step down that surface:

with the learning rate. A classification head turns the last hidden vector into class probabilities with softmax:

where is the number of classes, so the output of a deep network is always a normalized distribution over the labels you care about.

Deep Learning vs Traditional Machine Learning

Depth also buys parameter efficiency. A chain of dense layers with widths holds

parameters, dominated by the products . A 4,096 to 1,024 to 1,024 to 1,024 to 10 stack therefore holds 4,195,328 + 1,049,600 + 1,049,600 + 10,250 = 6,304,778 weights, while a single linear map from 4,096 straight to 10 holds only 40,970. The deep stack composes four nonlinear maps; the linear one collapses to one matrix.

Neural Network Depth Visualization

History: From Perceptrons to Deep Learning

The Perceptron Era (1958)

The AI Winter (1970s-1980s)

Minsky and Papert (1969) proved that single-layer perceptrons cannot solve the XOR problem, leading to a decades-long decline in neural network research. The field entered an "AI winter" as funding dried up. Over booleans XOR needs a product term, which no straight line in the input plane can express:

One hidden layer with a nonlinearity realizes it, which is exactly the argument for depth: each extra layer multiplies the class of functions the network can carve out.

Timeline of Deep Learning

When to Use Deep Learning

Hardware Requirements

Hardware budgets start from arithmetic, not from feelings. One training step on a dense network costs roughly two multiply-accumulates per weight per sample:

A batch of 64 through a 4,096 to 1,024 to 1,024 to 10 stack is about 2 x 64 x (4,194,304 + 1,048,576 + 10,240) = 671,344,640 FLOPs per step. At 10 steps per second an A100 clears that comfortably, while a GTX 1060 at roughly 9 teraFLOPS FP32 spends a large fraction of its budget on the same work, which is why wall-clock epoch time scales almost linearly with the products .

Memory follows the same accounting. Holding parameters in FP32 while training with Adam reserves room for the weights, the gradients, and two running moments:

So the 6,304,778 parameter stack above needs about 101 MB before activations, and a 7 billion parameter model needs roughly 112 GB, which is why 80 GB A100 cards and optimizer sharding exist at all.

The Three Pillars of Deep Learning

The convergence of these three factors around 2012 triggered the deep learning revolution. Without any one of them, modern deep learning would not be possible.

Summary

• Deep learning uses multi-layer neural networks to learn hierarchical representations from data
• The field evolved from perceptrons (1958) through AI winter to the modern revolution (2012+)
• Three pillars enabled the revolution: big data, GPU computing, and algorithmic advances
• Deep learning excels with unstructured data (images, text, audio) and large datasets
• Hardware requirements include GPUs with sufficient VRAM and computational resources

Next: Math Foundations for Deep Learning

Frequently Asked Questions

How is deep learning different from machine learning?

Deep learning is a subset of machine learning where the features are learned rather than engineered. A classical pipeline fixes the representation (histograms, SIFT vectors, n-grams) and trains a shallow model on top; a deep network optimizes the representation and the classifier jointly, so the same convolution weights that find edges at layer 1 also feed the object detectors at layer 20.

Why did deep learning only take off in 2012?

Three things converged. ImageNet made 1.2 million labeled images available, CUDA made GPUs programmable for matrix work, and ReLU plus better initialization made 8-layer networks trainable without vanishing gradients. Remove any one and AlexNet's 15.3 percent top-5 error would not have beaten the previous 26.2 percent result.

When is a neural network the wrong tool?

When your data is a tidy table with fewer than about 100,000 rows, when you need probabilities that are easy to calibrate and explain, or when compute is scarce. Gradient boosted trees trained in minutes usually win those fights, while a deep model of the same data needs hyperparameter search, augmentation, and a GPU to match it.