✦ScienexySCIENCE KNOWLEDGE LIBRARY
Science reference · SCI-CS-003-109

Gradient Descent and Its Variants

Gradient descent is an optimization method that updates model parameters in the direction opposite to the gradient of a loss function. Its variants differ in how they calculate and apply gradient updates.

θₜ₊₁ = θₜ − η∇J(θₜ)

Explanation

Gradient descent is an iterative optimization algorithm used to minimize an objective function, such as the loss of a machine learning model. It calculates the gradient of the objective with respect to trainable parameters and updates those parameters in the direction that most rapidly decreases the objective locally.

In batch gradient descent, the gradient is calculated using the entire training dataset for each update. This provides a gradient based on all training examples but can be computationally expensive for large datasets. Stochastic gradient descent (SGD) uses one randomly selected training example per update, producing frequent but noisy updates. Mini-batch gradient descent uses a small group of examples for each update and is widely used because it balances computational efficiency with gradient stability.

Several optimizers extend basic gradient descent. Momentum incorporates a fraction of previous updates to help accelerate progress in consistent directions and reduce oscillations. AdaGrad adapts learning rates for individual parameters based on accumulated squared gradients. RMSProp uses a moving average of squared gradients to adapt learning rates. Adam combines momentum-like first-moment estimates with second-moment estimates and is widely used in deep learning.

The learning rate controls the size of each update. If it is too large, optimization may become unstable or fail to converge. If it is too small, training may progress very slowly. Learning-rate schedules, warm-up strategies, and adaptive optimizers can help manage optimization behavior.

Gradient descent is commonly used together with backpropagation in neural networks. Backpropagation calculates gradients, while the optimizer uses them to update parameters.

Variables

SymbolMeaning

θₜ

Model parameters at iteration t

θₜ₊₁

Updated model parameters

η

Learning rate

∇J(θₜ)

Gradient of the objective function at the current parameters

J(θ)

Objective or loss function

Example or application

A neural network is trained using mini-batch gradient descent with a batch size of 32. The model processes 32 training examples, calculates the average gradient for that mini-batch, and updates its weights. This process repeats for successive mini-batches and across multiple epochs.

Text

Gradient descent minimizes a loss function by repeatedly updating model parameters in the direction opposite to the gradient. Batch gradient descent uses the full dataset for each update, stochastic gradient descent uses one example, and mini-batch gradient descent uses a small group of examples. Momentum, AdaGrad, RMSProp, and Adam modify the update process to improve optimization speed or adapt learning rates.

Key Points

  • Gradient descent minimizes an objective function through repeated updates.

  • Parameters are updated opposite to the gradient direction.

  • Batch gradient descent uses the entire dataset per update.

  • SGD uses one training example per update.

  • Mini-batch gradient descent uses a small subset of examples.

  • Momentum can reduce oscillations and accelerate consistent updates.

  • AdaGrad, RMSProp, and Adam use adaptive learning-rate strategies.

  • Learning-rate selection strongly influences training stability and speed.

Applications

  • Training neural networks

  • Optimizing regression models

  • Training deep learning classifiers

  • Minimizing differentiable machine learning objectives

  • Large-scale data-driven optimization

Learning Outcomes

  • Explain how gradient descent updates parameters.

  • Compare batch, stochastic, and mini-batch gradient descent.

  • Describe the basic ideas behind Momentum, AdaGrad, RMSProp, and Adam.

  • Explain how learning rate affects optimization.

Practice Questions

  • Why are parameters updated in the direction opposite to the gradient?

  • How does mini-batch gradient descent differ from batch gradient descent?

  • What problem can occur when the learning rate is too large?

Related Concepts

  • Backpropagation

  • Loss Functions

  • Learning Rate

  • Adam Optimizer

  • Stochastic Gradient Descent

Common Mistakes

  • Confusing gradient calculation with gradient-based optimization.

  • Assuming a larger learning rate always speeds up successful training.

  • Treating Adam as a guarantee of finding a global minimum.

Sources and verification

Sources are shown as provided in the project data. Not every source has yet been independently verified. Please report errors or missing citations.

  • Goodfellow, I., Bengio, Y., and Courville, A. (2016). Deep Learning. MIT Press.
  • Kingma, D. P., and Ba, J. (2015). Adam: A Method for Stochastic Optimization. International Conference on Learning Representations.

Suggest a correction or source · More in Artificial Intelligence & Machine Learning