The bias-variance tradeoff is a central concept in machine learning that explains why neither an excessively simple nor an unnecessarily complex model is always ideal. Bias refers to error introduced when a model's assumptions are too restrictive to represent the true underlying relationship. A high-bias model may fail to capture important patterns and commonly underfits the data.
Variance describes how strongly a model's predictions change when it is trained on different samples of data from the same underlying distribution. A high-variance model may learn small fluctuations and noise in its training dataset. It can perform very well on the training data but produce unstable predictions on new observations, which is commonly associated with overfitting.
The tradeoff occurs because increasing model complexity can reduce bias while increasing variance. Reducing model complexity may lower variance but increase bias. The objective is to select a model that balances these sources of error and performs well on unseen data.
For a standard squared-error regression setting, expected prediction error can be decomposed into squared bias, variance, and irreducible noise under appropriate assumptions. Irreducible noise represents uncertainty in the data that cannot be eliminated simply by choosing a different model.
Model selection, regularization, more representative training data, and cross-validation can help manage this tradeoff. The best balance depends on the dataset, model family, and learning problem.
Bias-Variance Tradeoff
The bias-variance tradeoff describes the balance between errors caused by overly simple model assumptions and errors caused by excessive sensitivity to training data.
E[(Y − ŷ)²] = Bias(ŷ)² + Var(ŷ) + σ²
Explanation
Variables
| Symbol | Meaning |
|---|---|
E[(Y − ŷ)²] | Expected squared prediction error |
Bias(ŷ) | Systematic difference between the average prediction and the true value |
Var(ŷ) | Variance of predictions across different training samples |
σ² | Irreducible noise variance under the assumed data model |
Example or application
Consider predicting house prices using only house size. A simple linear model may fail to capture important effects such as location and property condition, producing high bias. A very complex model may fit nearly every training example but respond strongly to small changes in the dataset, producing high variance. A suitably regularized model with relevant features may provide a better balance.
Text
Bias measures systematic error caused by overly restrictive model assumptions. Variance measures how sensitive a model is to changes in its training data. A model with high bias is often too simple and underfits, while a model with high variance may memorize training examples and overfit. Increasing model complexity can reduce bias but increase variance. The goal is to find a balance that supports accurate predictions on unseen data.
Key Points
Bias is associated with systematic modeling errors.
Variance measures sensitivity to the particular training sample.
High bias commonly leads to underfitting.
High variance commonly contributes to overfitting.
Model complexity often influences both bias and variance.
Irreducible noise cannot be eliminated through model selection alone.
The objective is strong generalization rather than perfect training fit.
Applications
Selecting model complexity
Choosing between simple and complex algorithms
Designing regularization strategies
Comparing machine learning models
Improving generalization performance
Learning Outcomes
Define bias and variance.
Explain how model complexity affects both.
Relate high bias to underfitting and high variance to overfitting.
Describe strategies for balancing prediction error.
Practice Questions
What is the difference between bias and variance?
Why can a highly complex model have high variance?
How does regularization help manage the bias-variance tradeoff?
Related Concepts
Overfitting and Underfitting
Model Complexity
Regularization
Cross-Validation
Generalization Error
Common Mistakes
Assuming that reducing bias always improves generalization.
Treating variance as the same thing as prediction error.
Ignoring irreducible noise when analyzing model performance.
Sources and verification
Sources are shown as provided in the project data. Not every source has yet been independently verified. Please report errors or missing citations.
- Bishop, C. M. (2006). Pattern Recognition and Machine Learning. Springer.
- Hastie, T., Tibshirani, R., and Friedman, J. (2009). The Elements of Statistical Learning. Springer.
Suggest a correction or source · More in Artificial Intelligence & Machine Learning