Ensemble learning is a machine learning approach in which multiple models, often called base learners, work together to produce a combined prediction. The central idea is that a carefully designed group of models can overcome some limitations of individual learners. Ensembles are especially useful when their component models make different types of errors.
Bagging, or bootstrap aggregating, trains multiple models independently on different bootstrap samples of the training data. Their predictions are combined through voting for classification or averaging for regression. Random forests are a well-known bagging-based method that also introduces random feature selection when building decision trees. Bagging is particularly useful for reducing the variance of unstable learners.
Boosting trains models sequentially, with later learners focusing on errors or patterns that earlier learners did not handle well. AdaBoost adjusts the importance of training examples according to previous classification errors. Gradient boosting builds new learners to improve an objective, often by fitting residual-related gradients. Popular gradient-boosting implementations include XGBoost, LightGBM, and CatBoost.
Stacking combines different base models using a separate model called a meta-learner. The meta-learner learns how to combine the predictions of the base models. To avoid leakage, stacking commonly uses out-of-fold predictions when training the meta-learner.
Ensemble methods can improve predictive performance, but they also introduce tradeoffs. They may require more computation, storage, and maintenance than a single model. Combining highly similar models may provide limited benefit, while poorly designed ensembles can overfit or create data leakage. The choice of ensemble method depends on the dataset, model diversity, computational resources, and deployment requirements.
Ensemble Learning Methods
Ensemble learning combines predictions from multiple models to produce a final prediction that can be more accurate, stable, or robust than an individual model.
ŷ_ensemble = (1/M)Σₘ₌₁ᴹ ŷₘ (regression averaging)
Explanation
Variables
| Symbol | Meaning |
|---|---|
ŷ_ensemble | Final ensemble prediction |
M | Number of models in the ensemble |
ŷₘ | Prediction produced by model m |
Example or application
Three regression models predict the price of a house as 200,000, 210,000, and 205,000. A simple averaging ensemble combines their predictions to produce 205,000. If the models capture different patterns and make partly independent errors, their combined prediction may be more stable than relying on one model alone.
Text
Ensemble learning combines multiple models to produce a final prediction. Bagging trains models independently on bootstrap samples and combines their outputs, commonly reducing variance. Random forests are a popular bagging-based method. Boosting trains models sequentially so later learners improve on earlier errors. Stacking uses a meta-learner to combine predictions from different base models. Ensembles can improve accuracy and stability, but they may increase computational cost and complexity.
Key Points
Ensemble learning combines multiple base models.
Bagging trains models independently on bootstrap samples.
Random forests combine decision trees with bootstrap sampling and feature randomness.
Boosting trains learners sequentially to improve predictions.
AdaBoost and gradient boosting are important boosting approaches.
Stacking uses a meta-learner to combine base-model predictions.
Model diversity can make ensemble predictions more useful.
Ensembles may increase computational and deployment complexity.
Applications
Credit scoring and risk prediction
Fraud detection
Medical prediction systems
Customer behavior prediction
Sales forecasting
Machine learning competitions
Image and text classification
Learning Outcomes
Explain the principle of ensemble learning.
Distinguish bagging, boosting, and stacking.
Describe how random forests use ensemble learning.
Identify the advantages and limitations of combining multiple models.
Practice Questions
What is ensemble learning?
How does bagging differ from boosting?
What role does a meta-learner play in stacking?
Why can combining diverse models improve prediction performance?
Related Concepts
Decision Trees
Random Forests
AdaBoost
Gradient Boosting
Model Diversity
Bootstrap Sampling
Common Mistakes
Assuming that adding more models always improves ensemble performance.
Confusing bagging with sequential boosting.
Training a stacking meta-learner using predictions that leak information from its training examples.
Ignoring the computational cost of an ensemble.
Sources and verification
Sources are shown as provided in the project data. Not every source has yet been independently verified. Please report errors or missing citations.
- Hastie, T., Tibshirani, R., and Friedman, J. (2009). The Elements of Statistical Learning. Springer.
- Dietterich, T. G. (2000). Ensemble Methods in Machine Learning. Multiple Classifier Systems.
- Breiman, L. (2001). Random Forests. Machine Learning.
Suggest a correction or source · More in Artificial Intelligence & Machine Learning