Activation Functions - Sigmoid, ReLU, GELU and The Dead Neuron Problem

Activation functions introduce non-linearity into neural networks, enabling them to learn complex patterns. Without them, a deep network would collapse into a linear model.

Why Activation Functions?

Sigmoid

The derivative follows from the quotient rule and never leaves the box shown below:

Carrying that factor through depth is the whole story of the vanishing gradient with sigmoids:

Ten sigmoid layers can shrink a gradient by six orders of magnitude before the first weight ever sees it. Sigmoid also saturates: for the output sits within 1 percent of its bound and the gradient is effectively zero, so saturated units stop learning entirely.

Tanh

Tanh is a shifted sigmoid and inherits a similar derivative shape with twice the peak:

Zero centering matters as much as the peak. Sigmoid outputs are always positive, so the gradients on a weight matrix arrive with the same sign and the update zig-zags along the coordinate axes. Tanh outputs average to zero near the origin, which keeps the gradient directions balanced, and that is the practical reason tanh outperformed sigmoid in hidden layers for two decades.

Activation Function Comparison

ReLU and Variants

ReLU changes the gradient story completely, because its derivative is the Heaviside step:

For positive units the gradient passes through at full strength no matter how deep the stack, which is why ReLU networks with hundreds of layers train while sigmoid networks stall near ten. The price is the kink at zero: the function is not differentiable exactly at , implementations pick a subgradient (usually 0), and any unit driven permanently negative becomes dead.

import numpy as np

def relu(x):
    return np.maximum(0.0, x)

def relu_grad(x):
    return (x > 0).astype(float)

x = np.array([-2.0, -0.5, 0.0, 0.5, 3.0])
print("relu:", relu(x))            # [0. 0. 0. 0.5 3. ]
print("grad:", relu_grad(x))       # [0. 0. 0. 1. 1.]
# Chain a gradient back: incoming grad [1, 1, 1, 1, 1] becomes [0, 0, 0, 1, 1]

The zero pattern of relu_grad is exactly what sparsity looks like: with batch inputs, a healthy ReLU layer keeps roughly 50 to 80 percent of its units firing on any given step.

GELU and Swish

Softmax

The Jacobian of softmax is dense but compact, and it is what the backward pass actually multiplies:

Multiplying it by the cross-entropy gradient collapses to , one subtraction per class, which is why frameworks fuse softmax and cross-entropy into a single fused kernel instead of forming this matrix.

When to Use Each Activation

The Dead Neuron Problem

Formally, a unit is dead when no sample in the training set can push it positive:

Zero gradient means the optimizer never revises the weights, so the unit cannot resurrect itself. He initialization keeps initial weights at variance so that about half the units start positive, Leaky ReLU gives the negative side a slope of so the gradient is never exactly zero, and batch normalization re-centers pre-activations every step. Any one of the three is usually enough; using all three is standard in vision backbones.

Summary

• Activation functions introduce non-linearity, enabling deep networks to learn complex patterns
• ReLU is the default for hidden layers: fast, prevents vanishing gradients, but causes dead neurons
• GELU/Swish are preferred for transformers and deep networks: smooth, self-gated
• Softmax for multi-class output, Sigmoid for binary output, None for regression
• Dead neurons are a practical concern with ReLU; use Leaky ReLU or careful initialization to mitigate

Next: Loss Functions for Deep Learning

Frequently Asked Questions

Why does non-linearity buy so much capacity?

A stack of linear maps is still one linear map, so its decision boundary is a single hyperplane no matter the depth. One nonlinearity per layer lets the network bend the input space, carve disconnected regions, and approximate piecewise functions: ReLU units in one layer already give linear pieces, and stacking them multiplies the arrangement possibilities exponentially in depth.

Why is ReLU's non-differentiability at zero not a problem in practice?

A single point carries zero measure in the input distribution, so stochastic gradient descent hits it with probability zero; implementations define the derivative there as 0 or 1 and move on. What matters is the surrounding region: inputs far negative give zero gradient forever, which is the dead neuron issue rather than the technicality at the origin.

When should an output layer have no activation?

Regression heads that predict an unbounded quantity use a raw linear output, because sigmoid or softmax would clip the range to 0 to 1 or a probability simplex. If the target itself is bounded, rescale the target instead of squashing the network, and only reach for sigmoid or softmax when the output genuinely has to be a probability.