✦ScienexySCIENCE KNOWLEDGE LIBRARY
Science reference · SCI-CS-003-108

Backpropagation

Backpropagation is an algorithm for efficiently calculating gradients of a neural network's loss with respect to its weights and biases by applying the chain rule of calculus.

δ⁽ˡ⁾ = ((W⁽ˡ⁺¹⁾)ᵀδ⁽ˡ⁺¹⁾) ⊙ f′(z⁽ˡ⁾); ∂L/∂W⁽ˡ⁾ = δ⁽ˡ⁾(a⁽ˡ⁻¹⁾)ᵀ

Explanation

Backpropagation is the central gradient-calculation procedure used to train many neural networks. After a forward pass produces a prediction, a loss function measures how far that prediction is from the target. Backpropagation then calculates how each trainable parameter contributes to the loss.

The algorithm works backward through the network, starting from the output layer and moving toward earlier layers. It applies the chain rule to calculate derivatives efficiently by reusing intermediate gradient calculations. These derivatives indicate how small changes in weights and biases would affect the loss.

For each layer, backpropagation calculates an error signal and uses it to determine gradients for the layer's weights and biases. An optimizer such as stochastic gradient descent or Adam then uses these gradients to update the parameters.

Backpropagation is not itself an optimization algorithm. It calculates gradients; the optimizer uses those gradients to change model parameters. This distinction is important when understanding neural network training.

The method is widely used in feedforward networks, convolutional neural networks, and many other differentiable architectures. Training difficulties can arise from vanishing or exploding gradients, poor initialization, unsuitable learning rates, and numerical instability. Techniques such as suitable activation functions, normalization, residual connections, and gradient clipping can help in relevant settings.

Variables

SymbolMeaning

δ⁽ˡ⁾

Error or gradient signal at layer l

W⁽ˡ⁺¹⁾

Weight matrix of the next layer

f′(z⁽ˡ⁾)

Derivative of the activation function evaluated at the pre-activation values

⊙

Element-wise multiplication

L

Loss function

a⁽ˡ⁻¹⁾

Activation vector from the previous layer

Example or application

A neural network predicts that an image contains a cat with probability 0.40, while the correct label is cat. The loss function measures the prediction error. Backpropagation calculates gradients for the network's weights and biases, showing how they contributed to the loss. An optimizer then adjusts those parameters to improve future predictions.

Text

Backpropagation calculates gradients that show how a neural network's loss changes with respect to its weights and biases. It begins with the loss at the output and propagates gradient information backward through the network using the chain rule. An optimization algorithm then uses these gradients to update the parameters. Backpropagation calculates the gradients, while the optimizer performs the parameter updates.

Key Points

  • Backpropagation calculates gradients of the loss function.

  • It applies the chain rule of calculus.

  • Gradient information moves backward from output layers.

  • Gradients describe how parameters influence the loss.

  • An optimizer uses gradients to update weights and biases.

  • Backpropagation is different from gradient descent.

  • Vanishing and exploding gradients can make training difficult.

Applications

  • Training feedforward neural networks

  • Training convolutional neural networks

  • Training deep learning models

  • Optimizing image classifiers

  • Developing speech and language models

Learning Outcomes

  • Explain the purpose of backpropagation.

  • Describe how the chain rule is used to calculate gradients.

  • Distinguish gradient calculation from parameter optimization.

  • Identify common gradient-related training difficulties.

Practice Questions

  • Why does backpropagation move from output layers toward earlier layers?

  • What role does the chain rule play in backpropagation?

  • How is backpropagation different from gradient descent?

Related Concepts

  • Forward Propagation

  • Gradient Descent

  • Loss Functions

  • Chain Rule

  • Vanishing Gradients

Common Mistakes

  • Treating backpropagation and gradient descent as the same algorithm.

  • Assuming backpropagation directly changes weights without an optimizer.

  • Ignoring the effect of activation-function derivatives on gradient flow.

Sources and verification

Sources are shown as provided in the project data. Not every source has yet been independently verified. Please report errors or missing citations.

  • Rumelhart, D. E., Hinton, G. E., and Williams, R. J. (1986). Learning Representations by Back-Propagating Errors. Nature.
  • Goodfellow, I., Bengio, Y., and Courville, A. (2016). Deep Learning. MIT Press.

Suggest a correction or source · More in Artificial Intelligence & Machine Learning