Many machine learning models depend on several parameters simultaneously. A multivariable function describes how an output changes according to multiple input variables. A partial derivative measures the rate of change of that function with respect to one selected variable while the other variables are held constant. When a function has several input variables, its partial derivatives can be combined into a gradient vector. The gradient points in the direction of the steepest local increase of a differentiable scalar-valued function, while its negative points in the direction of steepest local decrease under the usual Euclidean geometry. In machine learning, the gradient of a loss function describes how the loss changes with respect to model parameters. Gradient descent uses the negative gradient to update parameters in an attempt to reduce the loss. The gradient can be calculated using the chain rule, which is particularly important in neural networks and backpropagation. Partial derivatives and gradients therefore provide essential mathematical tools for understanding how multiple model parameters interact during training.
Partial Derivatives and Gradients
An explanation of partial derivatives and gradient vectors for analyzing multivariable functions and optimizing machine learning models.
∇f(x₁, x₂, ..., xₙ) = [∂f/∂x₁, ∂f/∂x₂, ..., ∂f/∂xₙ]
Explanation
Variables
| Symbol | Meaning |
|---|---|
∇f | The gradient vector of function f. |
f | A scalar-valued function of multiple variables. |
x₁, x₂, ..., xₙ | The independent variables of the function. |
∂f/∂xᵢ | The partial derivative of f with respect to variable xᵢ. |
n | The number of independent variables. |
Example or application
Consider f(x, y) = x² + 3y². Its partial derivatives are ∂f/∂x = 2x and ∂f/∂y = 6y. Therefore, its gradient is ∇f = [2x, 6y]. At the point (1, 2), the gradient is [2, 12].
Text
A partial derivative measures how a multivariable function changes with respect to one variable while the other variables are held fixed. The gradient is a vector containing all first-order partial derivatives of a scalar-valued function. It identifies the direction of steepest local increase, and its negative indicates the direction of steepest local decrease. Machine learning algorithms use gradients to update model parameters and reduce prediction loss.
Key Points
Partial derivatives apply to functions with multiple independent variables.
A partial derivative changes one variable while holding the others constant.
The gradient combines first-order partial derivatives into a vector.
The gradient indicates the direction of steepest local increase.
The negative gradient indicates the direction of steepest local decrease under Euclidean geometry.
The gradient's magnitude describes the strength of the local rate of increase.
Gradient descent uses gradient information to update model parameters.
The chain rule supports gradient calculations in complex models.
Backpropagation efficiently calculates gradients in neural networks.
Applications
Gradient descent
Backpropagation in neural networks
Multivariable loss optimization
Sensitivity analysis
Parameter estimation
Deep learning model training
Scientific computing
Learning Outcomes
Define a partial derivative.
Explain why other variables are held constant during partial differentiation.
Construct a gradient vector from partial derivatives.
Interpret the direction of a gradient.
Explain how gradients support machine learning optimization.
Practice Questions
What is a partial derivative?
How does a partial derivative differ from an ordinary derivative?
What information does a gradient vector provide?
Find the gradient of f(x, y) = x² + 3y².
Why does gradient descent use the negative gradient?
Related Concepts
Calculus and Derivatives in Machine Learning
Optimization and Loss Minimization
Gradient Descent
Chain Rule
Backpropagation
Common Mistakes
Changing multiple variables when calculating a partial derivative.
Confusing a gradient vector with a single partial derivative.
Assuming the gradient always points toward decreasing function values.
Ignoring the geometry and coordinate system when interpreting steepest directions.
Confusing gradient calculation with the parameter-update rule itself.
Sources and verification
Sources are shown as provided in the project data. Not every source has yet been independently verified. Please report errors or missing citations.
- Stewart, J. Calculus: Early Transcendentals.
- Deisenroth, M. P., Faisal, A. A., and Ong, C. S. Mathematics for Machine Learning.
- Goodfellow, I., Bengio, Y., and Courville, A. Deep Learning.
Suggest a correction or source · More in Artificial Intelligence & Machine Learning