✦ScienexySCIENCE KNOWLEDGE LIBRARY
Science reference · SCI-CS-003-077

Random Forest

Random forest is an ensemble learning algorithm that combines multiple decision trees to improve predictive performance and reduce the instability of individual trees.

Classification: ŷ = mode(T₁(x), T₂(x), ..., Tₘ(x)); Regression: ŷ = (1/m) Σⱼ₌₁ᵐ Tⱼ(x)

Explanation

Random forest is an ensemble method based on bagging, or bootstrap aggregation. Each tree is trained using a bootstrap sample drawn from the training dataset. During tree construction, a random subset of features is considered at each split, which encourages diversity among the trees. For classification, the forest commonly predicts the class receiving the most votes. For regression, it commonly averages the predictions of individual trees. Combining diverse trees often reduces variance compared with relying on a single decision tree. Random forests can model nonlinear relationships and interactions, require relatively limited feature scaling, and provide estimates of feature importance. However, they may require substantial memory and computation, and their predictions are generally less interpretable than those of a single small tree.

Variables

SymbolMeaning

Tⱼ(x)

Prediction from decision tree j

m

Number of trees in the forest

x

Input observation

ŷ

Combined prediction

Example or application

A forest containing 100 decision trees predicts whether a transaction is fraudulent. If 78 trees vote for fraud and 22 vote for legitimate, the classification output is fraud.

Text

Random forest builds many decision trees using randomized training samples and feature subsets, then combines their predictions.

Key Points

  • Random forest combines multiple decision trees.

  • It uses bootstrap samples during tree construction.

  • Random feature selection increases tree diversity.

  • Classification commonly uses majority voting.

  • Regression commonly uses averaging.

  • Ensembling can reduce prediction variance.

Applications

  • Fraud detection

  • Medical diagnosis

  • Credit scoring

  • Customer behavior prediction

  • Environmental forecasting

Learning Outcomes

  • Explain the principle of random forests.

  • Describe bootstrap aggregation.

  • Understand why random feature selection is used.

  • Distinguish random forest classification from regression.

Practice Questions

  • What is an ensemble learning algorithm?

  • What is bootstrap aggregation?

  • Why does random forest select random feature subsets?

  • How are predictions combined in classification and regression?

Related Concepts

  • Decision Trees

  • Ensemble Learning

  • Bagging

  • Bootstrap Sampling

  • Feature Importance

Common Mistakes

  • Assuming all trees are trained on exactly the same sample.

  • Confusing random forests with gradient boosting.

  • Assuming feature importance proves a causal relationship.

  • Ignoring computation and memory costs for large forests.

Sources and verification

Sources are shown as provided in the project data. Not every source has yet been independently verified. Please report errors or missing citations.

  • Breiman, L. (2001). Random Forests.
  • Hastie, T., Tibshirani, R., and Friedman, J. (2009). The Elements of Statistical Learning.

Suggest a correction or source · More in Artificial Intelligence & Machine Learning