✦ScienexySCIENCE KNOWLEDGE LIBRARY
Science reference · SCI-CS-003-092

Loss Functions and Cost Functions

Loss functions measure the error associated with a model's prediction, while cost functions commonly represent an aggregate of losses over a dataset. They provide objective measures that optimization algorithms attempt to minimize.

MSE = (1/n) Σᵢ₌₁ⁿ (yᵢ − ŷᵢ)²

Explanation

Loss functions and cost functions are fundamental components of machine learning optimization. A loss function evaluates how different a model's prediction is from the correct target for an individual training example. A cost function commonly aggregates these individual losses across a collection of examples, often by calculating their mean or sum. Terminology can vary between textbooks and machine learning frameworks, and the two terms are sometimes used interchangeably.

The choice of loss function depends on the learning task. Mean squared error is commonly used for regression because it calculates the average squared difference between predicted and actual numerical values. Mean absolute error calculates the average absolute difference and is less sensitive to large errors than squared error. Binary cross-entropy is frequently used for binary classification, while categorical cross-entropy is commonly used for multiclass classification.

During training, an optimization algorithm attempts to find model parameters that minimize the selected objective. Gradient-based methods calculate how the objective changes with respect to the model parameters and use this information to update them.

A loss function must be selected carefully because it influences what the model learns to prioritize. For example, squared error gives large errors a stronger influence because the differences are squared. In classification, cross-entropy penalizes predictions that assign low probability to the correct class. Minimizing a training objective is important, but the final model must also be evaluated on independent data.

Variables

SymbolMeaning

MSE

Mean squared error

n

Number of observations

yᵢ

Actual target value of observation i

ŷᵢ

Predicted target value of observation i

Σ

Summation across all observations

Example or application

Suppose a model predicts the prices of three houses as 200,000, 250,000, and 300,000, while their actual prices are 210,000, 240,000, and 310,000. The squared differences are 100,000,000, 100,000,000, and 100,000,000. The mean squared error is therefore 100,000,000 in squared currency units.

Text

A loss function measures the difference between a model's prediction and the correct answer for an individual example. A cost function commonly combines losses across multiple examples to evaluate the model's overall training objective. Regression models often use mean squared error or mean absolute error, while classification models frequently use cross-entropy. Optimization algorithms attempt to minimize the selected objective by adjusting model parameters. The choice of loss function affects how prediction errors influence training.

Key Points

  • A loss function measures prediction error for an example.

  • A cost function commonly aggregates losses across a dataset.

  • Mean squared error is widely used in regression.

  • Mean absolute error measures average absolute prediction error.

  • Cross-entropy is widely used in classification.

  • The optimization algorithm attempts to minimize the chosen objective.

  • A lower training loss does not automatically guarantee better test performance.

Applications

  • Training linear and nonlinear regression models

  • Optimizing neural networks

  • Training binary and multiclass classifiers

  • Comparing model prediction errors

  • Developing machine learning optimization procedures

Learning Outcomes

  • Explain the purpose of loss and cost functions.

  • Distinguish individual-example loss from an aggregated objective.

  • Identify common loss functions for regression and classification.

  • Explain how loss functions guide model optimization.

Practice Questions

  • What is the difference between a loss function and a cost function?

  • Why is mean squared error commonly used in regression?

  • Which type of loss function is commonly used for binary classification?

Related Concepts

  • Model Training

  • Gradient Descent

  • Mean Squared Error

  • Cross-Entropy

  • Optimization Algorithms

Common Mistakes

  • Assuming every machine learning task should use mean squared error.

  • Ignoring the units of squared-error measurements.

  • Treating training loss as a complete measure of generalization performance.

Sources and verification

Sources are shown as provided in the project data. Not every source has yet been independently verified. Please report errors or missing citations.

  • Bishop, C. M. (2006). Pattern Recognition and Machine Learning. Springer.
  • Goodfellow, I., Bengio, Y., and Courville, A. (2016). Deep Learning. MIT Press.

Suggest a correction or source · More in Artificial Intelligence & Machine Learning