Loss functions and cost functions are fundamental components of machine learning optimization. A loss function evaluates how different a model's prediction is from the correct target for an individual training example. A cost function commonly aggregates these individual losses across a collection of examples, often by calculating their mean or sum. Terminology can vary between textbooks and machine learning frameworks, and the two terms are sometimes used interchangeably.
The choice of loss function depends on the learning task. Mean squared error is commonly used for regression because it calculates the average squared difference between predicted and actual numerical values. Mean absolute error calculates the average absolute difference and is less sensitive to large errors than squared error. Binary cross-entropy is frequently used for binary classification, while categorical cross-entropy is commonly used for multiclass classification.
During training, an optimization algorithm attempts to find model parameters that minimize the selected objective. Gradient-based methods calculate how the objective changes with respect to the model parameters and use this information to update them.
A loss function must be selected carefully because it influences what the model learns to prioritize. For example, squared error gives large errors a stronger influence because the differences are squared. In classification, cross-entropy penalizes predictions that assign low probability to the correct class. Minimizing a training objective is important, but the final model must also be evaluated on independent data.
Loss Functions and Cost Functions
Loss functions measure the error associated with a model's prediction, while cost functions commonly represent an aggregate of losses over a dataset. They provide objective measures that optimization algorithms attempt to minimize.
MSE = (1/n) Σᵢ₌₁ⁿ (yᵢ − ŷᵢ)²
Explanation
Variables
| Symbol | Meaning |
|---|---|
MSE | Mean squared error |
n | Number of observations |
yᵢ | Actual target value of observation i |
ŷᵢ | Predicted target value of observation i |
Σ | Summation across all observations |
Example or application
Suppose a model predicts the prices of three houses as 200,000, 250,000, and 300,000, while their actual prices are 210,000, 240,000, and 310,000. The squared differences are 100,000,000, 100,000,000, and 100,000,000. The mean squared error is therefore 100,000,000 in squared currency units.
Text
A loss function measures the difference between a model's prediction and the correct answer for an individual example. A cost function commonly combines losses across multiple examples to evaluate the model's overall training objective. Regression models often use mean squared error or mean absolute error, while classification models frequently use cross-entropy. Optimization algorithms attempt to minimize the selected objective by adjusting model parameters. The choice of loss function affects how prediction errors influence training.
Key Points
A loss function measures prediction error for an example.
A cost function commonly aggregates losses across a dataset.
Mean squared error is widely used in regression.
Mean absolute error measures average absolute prediction error.
Cross-entropy is widely used in classification.
The optimization algorithm attempts to minimize the chosen objective.
A lower training loss does not automatically guarantee better test performance.
Applications
Training linear and nonlinear regression models
Optimizing neural networks
Training binary and multiclass classifiers
Comparing model prediction errors
Developing machine learning optimization procedures
Learning Outcomes
Explain the purpose of loss and cost functions.
Distinguish individual-example loss from an aggregated objective.
Identify common loss functions for regression and classification.
Explain how loss functions guide model optimization.
Practice Questions
What is the difference between a loss function and a cost function?
Why is mean squared error commonly used in regression?
Which type of loss function is commonly used for binary classification?
Related Concepts
Model Training
Gradient Descent
Mean Squared Error
Cross-Entropy
Optimization Algorithms
Common Mistakes
Assuming every machine learning task should use mean squared error.
Ignoring the units of squared-error measurements.
Treating training loss as a complete measure of generalization performance.
Sources and verification
Sources are shown as provided in the project data. Not every source has yet been independently verified. Please report errors or missing citations.
- Bishop, C. M. (2006). Pattern Recognition and Machine Learning. Springer.
- Goodfellow, I., Bengio, Y., and Courville, A. (2016). Deep Learning. MIT Press.
Suggest a correction or source · More in Artificial Intelligence & Machine Learning