✦ScienexySCIENCE KNOWLEDGE LIBRARY
Science reference · SCI-CS-003-106

Activation Functions

Activation functions transform the weighted input of a neuron into its output and introduce nonlinear behavior that allows neural networks to learn complex relationships.

Sigmoid: σ(x) = 1/(1 + e⁻ˣ); ReLU: f(x) = max(0, x); Tanh: f(x) = tanh(x); Softmax: pᵢ = eᶻⁱ / Σⱼeᶻʲ

Explanation

An activation function determines the output of an artificial neuron after its inputs have been combined using weights and a bias. It is applied to the neuron's pre-activation value and controls how information is passed to the next layer.

Activation functions are important because they introduce nonlinearity. Without nonlinear activation functions, a network made only of linear transformations is mathematically equivalent to a single linear transformation, regardless of how many such layers are stacked. Nonlinear functions allow neural networks to represent more complex relationships and decision boundaries.

Common activation functions include sigmoid, tanh, ReLU, and softmax. Sigmoid maps real-valued inputs to values between zero and one and is often used for binary classification outputs. Tanh maps inputs between negative one and one. ReLU returns zero for negative inputs and the input itself for positive inputs; it is widely used in hidden layers. Softmax converts a vector of logits into a probability distribution across mutually exclusive classes.

No activation function is ideal for every situation. Sigmoid and tanh can suffer from vanishing gradients in saturated regions. ReLU can produce inactive neurons when their inputs remain negative. Softmax is generally used at the output of multiclass classifiers rather than as a hidden-layer activation in ordinary feedforward networks.

Variables

SymbolMeaning

x

Input or pre-activation value

e

Base of the natural logarithm

zᵢ

Logit for class i

pᵢ

Softmax probability assigned to class i

Σ

Summation across output classes

Example or application

A neural network classifies an image into one of three mutually exclusive categories: cat, dog, or bird. Its output layer produces three logits. Softmax transforms these logits into probabilities that sum to one, allowing the model to select the category with the highest predicted probability.

Text

An activation function transforms a neuron's weighted input into an output. Its most important role is to introduce nonlinearity, allowing neural networks to learn complex relationships. Common functions include sigmoid, tanh, ReLU, and softmax. Sigmoid is often used for binary classification outputs, ReLU is common in hidden layers, and softmax is commonly used to produce class probabilities for mutually exclusive multiclass classification.

Key Points

  • Activation functions transform neuron inputs into outputs.

  • Nonlinear activations allow networks to model complex relationships.

  • Sigmoid produces outputs between zero and one.

  • Tanh produces outputs between negative one and one.

  • ReLU is widely used in hidden layers.

  • Softmax produces a probability distribution over classes.

  • Activation function selection affects gradient behavior and training.

Applications

  • Binary classification output layers

  • Multiclass classification

  • Hidden layers in deep neural networks

  • Image recognition

  • Language models

  • Regression networks using suitable output activations

Learning Outcomes

  • Explain why activation functions are necessary.

  • Describe the behavior of sigmoid, tanh, ReLU, and softmax.

  • Identify suitable activation functions for common network outputs.

  • Explain how activation functions influence neural network training.

Practice Questions

  • Why are nonlinear activation functions important in deep neural networks?

  • What is the output range of sigmoid and tanh?

  • Why is softmax commonly used for mutually exclusive multiclass classification?

Related Concepts

  • Artificial Neurons

  • Perceptrons

  • Forward Propagation

  • Backpropagation

  • Vanishing Gradients

Common Mistakes

  • Assuming a network of only linear layers can learn arbitrary nonlinear relationships.

  • Using softmax when independent multilabel outputs are required.

  • Ignoring saturation and gradient behavior when choosing activation functions.

Sources and verification

Sources are shown as provided in the project data. Not every source has yet been independently verified. Please report errors or missing citations.

  • Goodfellow, I., Bengio, Y., and Courville, A. (2016). Deep Learning. MIT Press.
  • Nair, V., and Hinton, G. E. (2010). Rectified Linear Units Improve Restricted Boltzmann Machines. Proceedings of ICML.

Suggest a correction or source · More in Artificial Intelligence & Machine Learning