Deep networks change data during the forward pass. They send gradients backward during the backward pass. Because of this, three design choices are closely connected: activation functions, gradient flow, and weight initialization. This guide explains that connection step by step, with experiments that you can run.

import numpy as np
import matplotlib.pyplot as plt

plt.style.use("default")
rng = np.random.default_rng(42)
x = np.linspace(-6, 6, 1000)

def plot_activation(x, y, name, formula, ylim=None):
    fig, ax = plt.subplots(figsize=(7, 4))
    ax.plot(x, y, linewidth=2.5)
    ax.axhline(0, color="black", linewidth=0.8)
    ax.axvline(0, color="black", linewidth=0.8)
    ax.grid(alpha=0.3)
    ax.set(xlabel="Input, x", ylabel="Output, f(x)", title=name)
    ax.text(0.03, 0.92, formula, transform=ax.transAxes, fontsize=12)
    if ylim is not None:
        ax.set_ylim(*ylim)
    plt.show()

Part I — Common activation functions

A layer first computes $z=W x+b$. It then applies an activation $a=f(z)$. Without nonlinear activations, many stacked layers are equal to one linear transformation. The activation also sets the local derivative used during backpropagation. It therefore affects how well gradients move through a deep network.

1. Sigmoid

Sigmoid maps every real number to $(0,1)$. Its derivative is never larger than $0.25$. It becomes close to zero at both ends of the curve. These flat parts are called the saturated regions.

Pros: smooth, limited to $(0,1)$, and easy to read as a probability for a yes/no event.
Cons: has flat regions that can cause vanishing gradients; not centered around zero.
Typical use: binary-classification outputs, independent multi-label outputs, and gates in LSTMs/GRUs—not usually deep hidden layers.

sigmoid = 1 / (1 + np.exp(-x))
plot_activation(x, sigmoid, "Sigmoid", r"$\sigma(x)=1/(1+e^{-x})$", (-0.1, 1.1))

2. Tanh

Tanh maps inputs to $(-1,1)$. Its output is centered around zero. However, its derivative becomes close to zero at both ends of the curve.

Pros: smooth, bounded, and zero-centered.
Cons: saturates for large $|x|$ and can cause vanishing gradients.
Typical use: candidate states in recurrent networks and outputs that must stay in a signed range; less common in modern deep feed-forward hidden layers.

tanh = np.tanh(x)
plot_activation(x, tanh, "Hyperbolic Tangent (tanh)", r"$f(x)=\tanh(x)$", (-1.2, 1.2))

3. ReLU

ReLU has derivative 1 for positive inputs and 0 for negative inputs. Its positive side does not become flat, so ReLU remains a strong starting choice.

Pros: fast to compute, often produces many zero outputs, and keeps gradients for positive inputs.
Cons: negative inputs have zero gradient and may create permanently inactive—or dead—neurons; output is not zero-centered.
Typical use: hidden layers in MLPs and CNNs, usually with He initialization.

relu = np.maximum(0, x)
plot_activation(x, relu, "ReLU", r"$f(x)=\max(0,x)$")

4. Leaky ReLU

Leaky ReLU keeps a small negative slope $\alpha$, commonly $0.01$ or $0.1$.

Pros: reduces the dead-ReLU problem and is nearly as cheap as ReLU.
Cons: the negative slope is another hyperparameter; activations are unbounded and not exactly zero-centered.
Typical use: a direct replacement for ReLU, especially in CNNs and GAN discriminators.

alpha = 0.1
leaky_relu = np.where(x >= 0, x, alpha * x)
plot_activation(x, leaky_relu, "Leaky ReLU (α = 0.1)", r"$f(x)=\max(\alpha x,x)$")

5. GELU

Here, $\Phi(x)$ is the standard normal cumulative distribution function (CDF). It is the area under the standard normal curve from $-\infty$ to $x$. In probability terms, $\Phi(x)=P(Z\leq x)$ for $Z\sim\mathcal N(0,1)$. GELU uses this value to smoothly control how much of the input passes through. It allows small negative outputs, and it decreases slightly over a small part of its curve.

Pros: smooth and works very well in large language and vision models.
Cons: needs more computation than ReLU; the improvement depends on the model and task.
Typical use: Transformer feed-forward blocks, BERT-style models, and Vision Transformers.

gelu = 0.5 * x * (1 + np.tanh(np.sqrt(2 / np.pi) * (x + 0.044715 * x**3)))
plot_activation(x, gelu, "GELU (tanh approximation)", r"$f(x)\approx x\Phi(x)$")

6. SiLU / Swish and SwiGLU

SiLU (also called Swish when $\beta=1$) applies one value at a time:

SwiGLU is not the same function. It is a gated feed-forward unit with two learned linear transformations. It is often written as

The first branch uses SiLU as a gate. The second branch carries the values. SwiGLU works on learned vectors, so the plot below shows only its SiLU gate.

Pros: SiLU is smooth and keeps small negative signals; SwiGLU uses the input to control its gate and works well in modern language models.
Cons: SiLU costs more than ReLU; SwiGLU needs extra linear transformations and parameters unless we reduce the hidden width.
Typical use: SiLU in modern CNNs, detectors, and diffusion models; SwiGLU in Transformer feed-forward blocks.

silu = x / (1 + np.exp(-x))
plot_activation(x, silu, "SiLU / Swish (the gate used by SwiGLU)", r"$f(x)=x\sigma(x)$")

Activation summary

curves = {"Sigmoid": sigmoid, "Tanh": tanh, "ReLU": relu,
          "Leaky ReLU": leaky_relu, "GELU": gelu, "SiLU": silu}
fig, ax = plt.subplots(figsize=(9, 5))
for name, values in curves.items():
    ax.plot(x, values, linewidth=2, label=name)
ax.axhline(0, color="black", linewidth=0.8)
ax.axvline(0, color="black", linewidth=0.8)
ax.grid(alpha=0.3)
ax.set(xlabel="Input, x", ylabel="Output, f(x)", title="Common activations",
       xlim=(-4, 4), ylim=(-1.5, 4))
ax.legend(ncol=2)
plt.show()

Part II — Vanishing and exploding gradients

Begin with one neuron

Consider one neuron with input $h_0$, weight $w_1$, pre-activation $z_1$, and output $h_1$:

Suppose the loss is $\mathcal L(h_1)$. The chain rule tells us how a change in $h_0$ affects the loss:

The incoming gradient is multiplied by two values from this neuron: the activation slope $f’(z_1)$ and the weight $w_1$. For example, suppose the incoming gradient is 1, $w_1=0.8$, and $f’(z_1)=0.25$. The gradient passed to $h_0$ is then $1\times0.25\times0.8=0.2$.

Extend the chain rule through many scalar layers

Now consider a chain of $L$ neurons, with one number at each layer. For layer $l$, $z_l=w_lh_{l-1}+b_l$ and $h_l=f(z_l)$. Applying the same rule at every layer gives

Each layer has an exact multiplier $w_lf’(z_l)$. We do not need to call it a “typical factor.” If every multiplier were $0.8$, a gradient of 1 would become $0.8^{20}\approx0.0115$ after 20 layers. If every multiplier were $1.2$, it would become $1.2^{20}\approx38.3$. In a real network, the multipliers change across layers and input examples. Their full product still determines the final gradient.

• Vanishing gradient: the product becomes very small. Early layers then receive almost no signal for learning.
• Exploding gradient: the product becomes very large. Weight updates become unstable and may produce infinite values or NaNs.

One multiplier below 1 does not guarantee a vanishing gradient. In the same way, one multiplier above 1 does not guarantee an exploding gradient. The full product matters. For example, $2\times0.1=0.2$, so one small multiplier can be stronger than one large multiplier.

From scalar neurons to vector layers

A real dense layer uses vectors instead of single numbers. It also uses a weight matrix:

Let $g_l=\partial\mathcal L/\partial h_l$ be the gradient that arrives at layer $l$. Backpropagation through this layer gives

Here, $W_l^\top$ sends the gradient backward through the linear transformation. $D_l$ is a diagonal matrix. Its diagonal contains one activation derivative for each neuron. Applying this equation to every layer gives

The order matters because changing the order of matrix multiplication usually changes the result.

For the forward pass, the layer Jacobian is $J_l=D_lW_l$. A Jacobian describes how a small change in the input changes the output. We use the spectral norm to measure the largest stretching effect of this matrix. Its definition is

In simple words, try every vector $v$ of length 1, multiply it by $J_l$, and measure the new length. The largest possible new length is the spectral norm. For example, if $J=\operatorname{diag}(2,0.5)$, the matrix stretches one direction by 2 and the other by 0.5. Therefore, $|J|_2=2$. The spectral norm is also the largest singular value of the matrix.

This definition gives the following upper bound for the gradient:

If the product of these layer norms becomes very small, the gradient must become small. If the product becomes very large, the gradient may explode. This is only an upper bound, so it does not give the exact gradient. Matrices can change the gradient direction, and some effects can cancel each other. Still, the spectral norm gives us a clear and exact way to discuss how strongly each layer can stretch a gradient.

A numerical two-layer example

Consider a network with two tanh layers and one number in each layer. Let the weights be $w_1=0.5$ and $w_2=0.8$. Let the values before the activations be $z_1=1$ and $z_2=2$. Assume that the gradient from the loss is $\partial\mathcal L/\partial h_2=1$. Since $\tanh’(z)=1-\tanh^2(z)$,

Moving backward through one layer at a time gives

After only two tanh layers in their flat regions, the gradient has lost almost 99% of its size. Both the activation slopes and the weights made it smaller.

incoming_gradient = 1.0
weights = [0.8, 0.5]          # traversed backward: layer 2, then layer 1
pre_activations = [2.0, 1.0]
gradient = incoming_gradient
print(f"Gradient at output: {gradient:.6f}")
for layer, (weight, z_value) in enumerate(zip(weights, pre_activations), start=1):
    derivative = 1 - np.tanh(z_value) ** 2
    gradient *= weight * derivative
    print(f"After backward step {layer}: {gradient:.6f} "
          f"(weight={weight}, tanh'={derivative:.6f})")

Why sigmoid and tanh often cause vanishing gradients

For sigmoid, $\sigma’(z)=\sigma(z)(1-\sigma(z))$, so $0<\sigma’(z)\leq0.25$. The largest derivative occurs at $z=0$. Even there, the activation multiplies the gradient by only $0.25$ per layer. Across 20 layers, $0.25^{20}\approx9.1\times10^{-13}$. At $z=5$, $\sigma’(5)\approx0.00665$, so the gradient becomes much smaller. Larger weights can partly balance this effect. However, they can also make the next values of $z$ larger and push sigmoid further into its flat regions.

For tanh, $f’(z)=1-\tanh^2(z)$. Its largest derivative is 1 at $z=0$. Tanh can therefore pass gradients well when its inputs stay near zero. However, $f’(2)\approx0.071$ and $f’(5)\approx0.00018$. When tanh outputs get close to $-1$ or $1$, the curve is almost flat and the gradients almost disappear.

ReLU has derivative 1 for $z>0$, so its positive side does not make the gradient smaller. For $z<0$, however, its derivative is 0, so that neuron completely blocks the gradient. Leaky ReLU replaces zero with a small slope $\alpha$. This keeps a small backward signal.

sigmoid_prime = sigmoid * (1 - sigmoid)
tanh_prime = 1 - tanh**2
relu_prime = (x > 0).astype(float)
leaky_prime = np.where(x >= 0, 1.0, 0.1)

fig, ax = plt.subplots(figsize=(8, 4.5))
for name, derivative in {"Sigmoid": sigmoid_prime, "Tanh": tanh_prime,
                         "ReLU": relu_prime, "Leaky ReLU": leaky_prime}.items():
    ax.plot(x, derivative, linewidth=2, label=name)
ax.set(xlabel="Pre-activation, x", ylabel="Derivative, f′(x)",
       title="Activation derivatives determine local gradient flow", ylim=(-0.05, 1.08))
ax.grid(alpha=0.3)
ax.legend()
plt.show()

A scalar chain-rule demonstration

depth = np.arange(1, 51)
factors = {
    "|w f′(z)| = 0.25 (sigmoid, |w| = 1, z = 0)": 0.25,
    "|w f′(z)| = 0.80 (contracts)": 0.80,
    "|w f′(z)| = 1.00 (preserves)": 1.00,
    "|w f′(z)| = 1.20 (expands)": 1.20,
}
fig, ax = plt.subplots(figsize=(8, 4.5))
for label, factor in factors.items():
    ax.semilogy(depth, factor**depth, linewidth=2, label=label)
ax.set(xlabel="Number of multiplied layers", ylabel="Gradient magnitude (log scale)",
       title="Repeated Jacobian factors shrink or amplify gradients")
ax.grid(alpha=0.3, which="both")
ax.legend()
plt.show()

Common remedies

Good weight initialization is the first defense, and it is the focus of Part III. In practice, we also use activations without flat positive regions, residual connections, normalization, and gated recurrent units. Gradient clipping is especially useful for exploding gradients. These methods help gradients move through the network, but we should still monitor the values of activations and gradients.

Part III — Weight initialization

If we set every weight to zero, all neurons in a layer learn the same features. We therefore start with random weights. The size of these random weights matters. Very small weights make signals and gradients smaller. Very large weights make them larger, or push sigmoid and tanh into their flat regions.

Assume that the inputs and weights are independent and have a mean of zero. We can then estimate their variance with

Variance tells us how spread out the values are. A good initialization chooses $\operatorname{Var}(W)$ so that signals do not quickly become too small or too large across many layers.

Xavier / Glorot initialization — suited to tanh

Xavier initialization tries to keep a similar variance in both the forward and backward passes. It works well with activations that are roughly symmetric around zero:

or, for a uniform distribution,

Here, $n_{\text{in}}$ is the number of inputs to the layer, and $n_{\text{out}}$ is the number of outputs. They are also called fan-in and fan-out. When they are equal, the weight variance is $1/n$. This helps keep tanh inputs near zero, where tanh is almost linear and has a large derivative. It reduces the chance that tanh immediately moves close to $-1$ or $1$. Xavier is also common with sigmoid, but it cannot change sigmoid’s maximum derivative of $0.25$.

He / Kaiming initialization — suited to ReLU

If the values before ReLU are symmetric around zero, ReLU sets about half of them to zero. He initialization makes up for this loss by using twice the weight variance:

A common uniform version uses values between $-\sqrt{6/n_{\text{in}}}$ and $+\sqrt{6/n_{\text{in}}}$. For layers with the same input and output width, Xavier uses a variance close to $1/n$, while He uses $2/n$. The extra factor of 2 helps because ReLU removes about half of the input values. He initialization is therefore the usual choice for ReLU and its variants. Deep-learning frameworks may provide a gain setting that changes the scale for a chosen Leaky ReLU slope.

Experiment: signal variance through depth

We pass a batch of data through random dense layers without bias terms. We record the activation variance after each layer. This is not a training process. It only shows the effect of the activation and initialization at the start. A good pair stops the variance from quickly becoming zero or extremely large.

def propagate_variance(activation, init, depth=40, width=256, batch=512, seed=0):
    local_rng = np.random.default_rng(seed)
    h = local_rng.standard_normal((batch, width))
    variances = [h.var()]
    for _ in range(depth):
        if init == "small":
            std = 0.01
        elif init == "xavier":
            std = np.sqrt(1 / width)  # equal fan-in and fan-out
        elif init == "he":
            std = np.sqrt(2 / width)
        else:
            raise ValueError("Unknown initialization")
        weights = local_rng.standard_normal((width, width)) * std
        h = activation(h @ weights)
        variances.append(h.var())
    return np.asarray(variances)

experiments = {
    "tanh + small": (np.tanh, "small"),
    "tanh + Xavier": (np.tanh, "xavier"),
    "tanh + He": (np.tanh, "he"),
    "ReLU + Xavier": (lambda z: np.maximum(0, z), "xavier"),
    "ReLU + He": (lambda z: np.maximum(0, z), "he"),
}

fig, ax = plt.subplots(figsize=(9, 5))
for label, (activation, init) in experiments.items():
    values = propagate_variance(activation, init)
    ax.semilogy(range(len(values)), values, linewidth=2, label=label)
ax.set(xlabel="Layer", ylabel="Activation variance (log scale)",
       title="Initialization controls signal scale through depth")
ax.grid(alpha=0.3, which="both")
ax.legend(ncol=2)
plt.show()

The curves use random values, so the result can change slightly between experiments. We should not expect the variance to stay exactly constant. Tanh’s flat regions, the limited layer width, activation means that are not zero, and our independence assumptions all affect the result. The main pattern is clear: weights that are too small destroy the signal, Xavier is a good match for symmetric activations such as tanh, and He provides the extra scale needed after ReLU removes negative values.

Practical recipe

Initialization controls the network only before training starts. The optimizer changes the weights as soon as training begins. Residual paths and normalization help modern deep networks remain trainable. Use Xavier and He as strong starting choices. Then check the activation variance, the number of values in flat regions, the number of dead neurons, and the gradient size in each layer.

Final takeaway

Choose activations and weight initialization together. Sigmoid and tanh can make gradients very small because their derivatives approach zero in the flat parts of their curves. ReLU has a derivative of 1 on its positive side, but it removes negative signals. Xavier initialization helps tanh stay near the useful center of its curve. He initialization makes up for the negative half that ReLU removes. These choices do not solve every training problem, but they give a deep network a much better starting point.