Overfitting and underfitting are two common problems in machine learning model development. They describe situations in which a model fails to achieve a suitable balance between learning the training data and generalizing to new observations.
Overfitting occurs when a model captures not only meaningful relationships but also random fluctuations, measurement errors, or accidental patterns in the training dataset. An overfitted model often has very low training error but significantly higher validation or test error. It may result from excessive model complexity, insufficient training data, weak regularization, or training for too long in some learning settings.
Underfitting occurs when a model is unable to learn important relationships in the data. Its structure may be too simple, its features may be inadequate, or it may not have been trained sufficiently. An underfitted model usually performs poorly on both training and validation data.
Several techniques can help address these problems. Overfitting may be reduced through regularization, data augmentation where appropriate, feature selection, early stopping, or collecting more representative data. Underfitting may be addressed by selecting a more expressive model, improving features, reducing excessive regularization, or training appropriately.
Learning curves, which compare training and validation performance as training progresses or data increases, can help diagnose these problems. The goal is to achieve a model that captures meaningful patterns without memorizing the training examples.
Overfitting and Underfitting
Overfitting occurs when a model learns training data too closely, including noise, while underfitting occurs when a model is too simple to capture important patterns. Both can reduce performance on unseen data.
Explanation
Example or application
A decision tree with very little depth may fail to distinguish meaningful customer groups, demonstrating underfitting. A decision tree that creates a separate branch for nearly every training observation may memorize the dataset and perform poorly on new customers, demonstrating overfitting.
Text
Overfitting happens when a model learns the training data too closely, including noise and accidental patterns. It usually produces excellent training performance but weaker validation or test performance. Underfitting happens when a model is too simple or insufficiently trained to capture important relationships, causing poor performance on both training and unseen data. Model selection, regularization, feature improvement, and suitable training procedures can help address these issues.
Key Points
Overfitting means learning training-specific details that do not generalize.
Underfitting means failing to learn important patterns.
Overfitting commonly produces low training error and higher validation error.
Underfitting commonly produces high training and validation errors.
Regularization and early stopping can help reduce overfitting.
More expressive models and better features can help address underfitting.
Validation data is important for detecting generalization problems.
Applications
Diagnosing decision tree performance
Training neural networks
Developing predictive regression models
Selecting machine learning model complexity
Improving model generalization
Learning Outcomes
Distinguish overfitting from underfitting.
Recognize their typical training and validation performance patterns.
Identify common causes of each problem.
Describe techniques for reducing both problems.
Practice Questions
How can you identify overfitting by comparing training and validation performance?
Why does an excessively simple model underfit?
Name two techniques that can help reduce overfitting.
Related Concepts
Bias-Variance Tradeoff
Regularization
Early Stopping
Learning Curves
Cross-Validation
Common Mistakes
Assuming a very low training error guarantees a good model.
Trying to solve underfitting only by adding more training data.
Using test data repeatedly to decide when to stop training.
Sources and verification
Sources are shown as provided in the project data. Not every source has yet been independently verified. Please report errors or missing citations.
- Hastie, T., Tibshirani, R., and Friedman, J. (2009). The Elements of Statistical Learning. Springer.
- Goodfellow, I., Bengio, Y., and Courville, A. (2016). Deep Learning. MIT Press.
Suggest a correction or source · More in Artificial Intelligence & Machine Learning