✦ScienexySCIENCE KNOWLEDGE LIBRARY
Science reference · SCI-CS-003-093

Bias-Variance Tradeoff

The bias-variance tradeoff describes the balance between errors caused by overly simple model assumptions and errors caused by excessive sensitivity to training data.

E[(Y − ŷ)²] = Bias(ŷ)² + Var(ŷ) + σ²

Explanation

The bias-variance tradeoff is a central concept in machine learning that explains why neither an excessively simple nor an unnecessarily complex model is always ideal. Bias refers to error introduced when a model's assumptions are too restrictive to represent the true underlying relationship. A high-bias model may fail to capture important patterns and commonly underfits the data.

Variance describes how strongly a model's predictions change when it is trained on different samples of data from the same underlying distribution. A high-variance model may learn small fluctuations and noise in its training dataset. It can perform very well on the training data but produce unstable predictions on new observations, which is commonly associated with overfitting.

The tradeoff occurs because increasing model complexity can reduce bias while increasing variance. Reducing model complexity may lower variance but increase bias. The objective is to select a model that balances these sources of error and performs well on unseen data.

For a standard squared-error regression setting, expected prediction error can be decomposed into squared bias, variance, and irreducible noise under appropriate assumptions. Irreducible noise represents uncertainty in the data that cannot be eliminated simply by choosing a different model.

Model selection, regularization, more representative training data, and cross-validation can help manage this tradeoff. The best balance depends on the dataset, model family, and learning problem.

Variables

SymbolMeaning

E[(Y − ŷ)²]

Expected squared prediction error

Bias(ŷ)

Systematic difference between the average prediction and the true value

Var(ŷ)

Variance of predictions across different training samples

σ²

Irreducible noise variance under the assumed data model

Example or application

Consider predicting house prices using only house size. A simple linear model may fail to capture important effects such as location and property condition, producing high bias. A very complex model may fit nearly every training example but respond strongly to small changes in the dataset, producing high variance. A suitably regularized model with relevant features may provide a better balance.

Text

Bias measures systematic error caused by overly restrictive model assumptions. Variance measures how sensitive a model is to changes in its training data. A model with high bias is often too simple and underfits, while a model with high variance may memorize training examples and overfit. Increasing model complexity can reduce bias but increase variance. The goal is to find a balance that supports accurate predictions on unseen data.

Key Points

  • Bias is associated with systematic modeling errors.

  • Variance measures sensitivity to the particular training sample.

  • High bias commonly leads to underfitting.

  • High variance commonly contributes to overfitting.

  • Model complexity often influences both bias and variance.

  • Irreducible noise cannot be eliminated through model selection alone.

  • The objective is strong generalization rather than perfect training fit.

Applications

  • Selecting model complexity

  • Choosing between simple and complex algorithms

  • Designing regularization strategies

  • Comparing machine learning models

  • Improving generalization performance

Learning Outcomes

  • Define bias and variance.

  • Explain how model complexity affects both.

  • Relate high bias to underfitting and high variance to overfitting.

  • Describe strategies for balancing prediction error.

Practice Questions

  • What is the difference between bias and variance?

  • Why can a highly complex model have high variance?

  • How does regularization help manage the bias-variance tradeoff?

Related Concepts

  • Overfitting and Underfitting

  • Model Complexity

  • Regularization

  • Cross-Validation

  • Generalization Error

Common Mistakes

  • Assuming that reducing bias always improves generalization.

  • Treating variance as the same thing as prediction error.

  • Ignoring irreducible noise when analyzing model performance.

Sources and verification

Sources are shown as provided in the project data. Not every source has yet been independently verified. Please report errors or missing citations.

  • Bishop, C. M. (2006). Pattern Recognition and Machine Learning. Springer.
  • Hastie, T., Tibshirani, R., and Friedman, J. (2009). The Elements of Statistical Learning. Springer.

Suggest a correction or source · More in Artificial Intelligence & Machine Learning