Forward propagation, also called the forward pass, is the computation through which a neural network transforms input data into a prediction. Information moves from the input layer through hidden layers and finally reaches the output layer.
At each layer, the network performs a weighted sum of the previous layer's activations, adds a bias, and applies an activation function. The resulting activation values become the inputs to the next layer. This process continues until the output layer produces a prediction.
For a simple feedforward network, the calculation at each layer can be represented using matrix operations. The weight matrix determines how strongly each previous-layer unit contributes to each current-layer unit. The bias vector shifts the weighted sums, while the activation function introduces the layer's nonlinear transformation.
During training, the predicted output is compared with the target using a loss function. The resulting loss is then used by backpropagation to calculate gradients for updating the network's parameters. Forward propagation itself calculates the prediction; it does not independently update the weights.
Forward propagation is also used during inference, when a trained model receives new input data and produces an output. During inference, parameter values normally remain fixed.
Forward Propagation
Forward propagation is the process of passing input data through the layers of a neural network to calculate its output or prediction.
z⁽ˡ⁾ = W⁽ˡ⁾a⁽ˡ⁻¹⁾ + b⁽ˡ⁾; a⁽ˡ⁾ = f⁽ˡ⁾(z⁽ˡ⁾)
Explanation
Variables
| Symbol | Meaning |
|---|---|
z⁽ˡ⁾ | Pre-activation vector at layer l |
W⁽ˡ⁾ | Weight matrix of layer l |
a⁽ˡ⁻¹⁾ | Activation vector from the previous layer |
b⁽ˡ⁾ | Bias vector of layer l |
f⁽ˡ⁾ | Activation function for layer l |
a⁽ˡ⁾ | Activation vector produced by layer l |
Example or application
An image is supplied to a trained neural network. Its pixel values enter the input layer, pass through hidden layers that extract visual patterns, and reach the output layer. The output layer produces probabilities for possible image categories, and the category with the highest probability becomes the prediction.
Text
Forward propagation sends input values through a neural network from the input layer toward the output layer. Each layer calculates weighted sums, adds biases, and applies activation functions. The final layer produces a prediction. During training, the prediction is evaluated using a loss function, while backpropagation calculates gradients for parameter updates. During inference, forward propagation is used to generate predictions without normally changing the learned parameters.
Key Points
Forward propagation computes a neural network's output.
Information moves from input layers toward output layers.
Each layer applies weights, biases, and activation functions.
Matrix operations make forward computation efficient.
The loss function evaluates the prediction during training.
Forward propagation does not itself update model parameters.
Inference also uses forward propagation with fixed learned parameters.
Applications
Generating predictions from trained neural networks
Image classification
Speech recognition
Language processing
Regression using neural networks
Calculating training loss
Learning Outcomes
Explain the forward propagation process.
Describe the computation performed by an individual layer.
Distinguish forward propagation from backpropagation.
Explain how forward propagation is used during training and inference.
Practice Questions
What happens during forward propagation?
Why does each layer use weights, biases, and activation functions?
Does forward propagation update the network's weights?
Related Concepts
Artificial Neural Networks
Activation Functions
Backpropagation
Loss Functions
Neural Network Training
Common Mistakes
Confusing forward propagation with backpropagation.
Assuming that predictions are automatically correct because they pass through many layers.
Thinking that the forward pass alone updates weights.
Sources and verification
Sources are shown as provided in the project data. Not every source has yet been independently verified. Please report errors or missing citations.
- Goodfellow, I., Bengio, Y., and Courville, A. (2016). Deep Learning. MIT Press.
- Nielsen, M. A. (2015). Neural Networks and Deep Learning. Determination Press.
Suggest a correction or source · More in Artificial Intelligence & Machine Learning