✦ScienexySCIENCE KNOWLEDGE LIBRARY
Science reference · SCI-CS-003-080

Gradient Boosting Algorithms

Gradient boosting is an ensemble learning technique that builds predictive models sequentially, with each new model attempting to reduce the errors of the existing ensemble.

Fₘ(x) = Fₘ₋₁(x) + ηhₘ(x)

Explanation

Gradient boosting is an ensemble method that constructs a model in stages. It begins with an initial prediction and repeatedly adds a new weak learner, often a shallow decision tree, to improve the current ensemble. Each new learner is fitted to the negative gradient of a chosen loss function with respect to the current predictions. For squared-error regression, this process is closely related to fitting residual errors. The learning rate controls how much each new learner contributes, while the number and depth of trees influence model complexity. Popular implementations include Gradient Boosting Machines, XGBoost, LightGBM, and CatBoost. Gradient boosting can achieve strong predictive performance on structured datasets, but it requires careful tuning and can overfit when model complexity is excessive. It is generally more sequential to train than bagging-based methods such as random forests.

Variables

SymbolMeaning

Fₘ(x)

Ensemble prediction after stage m

Fₘ₋₁(x)

Ensemble prediction before adding the new learner

hₘ(x)

New weak learner fitted to the negative loss gradient

η

Learning rate controlling the contribution of the new learner

m

Boosting iteration number

Example or application

A gradient boosting model predicts house prices. It first makes basic predictions, then adds shallow decision trees that progressively correct patterns in the remaining prediction errors until the ensemble achieves a better fit.

Text

Gradient boosting combines many weak predictive models, commonly decision trees, into a stronger model through iterative optimization.

Key Points

  • Gradient boosting builds learners sequentially.

  • Decision trees are commonly used as weak learners.

  • Each stage follows the negative gradient of the loss function.

  • The learning rate controls each learner's contribution.

  • Tree depth and iteration count affect model complexity.

  • Popular implementations include XGBoost, LightGBM, and CatBoost.

Applications

  • House price prediction

  • Fraud detection

  • Customer churn prediction

  • Credit risk modeling

  • Ranking and recommendation systems

Learning Outcomes

  • Explain the sequential nature of gradient boosting.

  • Understand how new learners improve an existing ensemble.

  • Describe the role of the learning rate.

  • Compare boosting with bagging-based ensemble methods.

Practice Questions

  • What is gradient boosting?

  • Why are weak learners added sequentially?

  • What does the learning rate control?

  • How does gradient boosting differ from random forest?

Related Concepts

  • Ensemble Learning

  • Decision Trees

  • Boosting

  • Gradient Descent

  • Loss Functions

  • Random Forest

Common Mistakes

  • Confusing gradient boosting with ordinary gradient descent.

  • Assuming every boosting implementation uses identical training procedures.

  • Using an excessively large learning rate.

  • Ignoring overfitting when increasing the number or depth of trees.

Sources and verification

Sources are shown as provided in the project data. Not every source has yet been independently verified. Please report errors or missing citations.

  • Friedman, J. H. (2001). Greedy Function Approximation: A Gradient Boosting Machine.
  • Chen, T., and Guestrin, C. (2016). XGBoost: A Scalable Tree Boosting System.

Suggest a correction or source · More in Artificial Intelligence & Machine Learning