Gradient boosting is an ensemble method that constructs a model in stages. It begins with an initial prediction and repeatedly adds a new weak learner, often a shallow decision tree, to improve the current ensemble. Each new learner is fitted to the negative gradient of a chosen loss function with respect to the current predictions. For squared-error regression, this process is closely related to fitting residual errors. The learning rate controls how much each new learner contributes, while the number and depth of trees influence model complexity. Popular implementations include Gradient Boosting Machines, XGBoost, LightGBM, and CatBoost. Gradient boosting can achieve strong predictive performance on structured datasets, but it requires careful tuning and can overfit when model complexity is excessive. It is generally more sequential to train than bagging-based methods such as random forests.
Gradient Boosting Algorithms
Gradient boosting is an ensemble learning technique that builds predictive models sequentially, with each new model attempting to reduce the errors of the existing ensemble.
Fₘ(x) = Fₘ₋₁(x) + ηhₘ(x)
Explanation
Variables
| Symbol | Meaning |
|---|---|
Fₘ(x) | Ensemble prediction after stage m |
Fₘ₋₁(x) | Ensemble prediction before adding the new learner |
hₘ(x) | New weak learner fitted to the negative loss gradient |
η | Learning rate controlling the contribution of the new learner |
m | Boosting iteration number |
Example or application
A gradient boosting model predicts house prices. It first makes basic predictions, then adds shallow decision trees that progressively correct patterns in the remaining prediction errors until the ensemble achieves a better fit.
Text
Gradient boosting combines many weak predictive models, commonly decision trees, into a stronger model through iterative optimization.
Key Points
Gradient boosting builds learners sequentially.
Decision trees are commonly used as weak learners.
Each stage follows the negative gradient of the loss function.
The learning rate controls each learner's contribution.
Tree depth and iteration count affect model complexity.
Popular implementations include XGBoost, LightGBM, and CatBoost.
Applications
House price prediction
Fraud detection
Customer churn prediction
Credit risk modeling
Ranking and recommendation systems
Learning Outcomes
Explain the sequential nature of gradient boosting.
Understand how new learners improve an existing ensemble.
Describe the role of the learning rate.
Compare boosting with bagging-based ensemble methods.
Practice Questions
What is gradient boosting?
Why are weak learners added sequentially?
What does the learning rate control?
How does gradient boosting differ from random forest?
Related Concepts
Ensemble Learning
Decision Trees
Boosting
Gradient Descent
Loss Functions
Random Forest
Common Mistakes
Confusing gradient boosting with ordinary gradient descent.
Assuming every boosting implementation uses identical training procedures.
Using an excessively large learning rate.
Ignoring overfitting when increasing the number or depth of trees.
Sources and verification
Sources are shown as provided in the project data. Not every source has yet been independently verified. Please report errors or missing citations.
- Friedman, J. H. (2001). Greedy Function Approximation: A Gradient Boosting Machine.
- Chen, T., and Guestrin, C. (2016). XGBoost: A Scalable Tree Boosting System.
Suggest a correction or source · More in Artificial Intelligence & Machine Learning