Random forest is an ensemble method based on bagging, or bootstrap aggregation. Each tree is trained using a bootstrap sample drawn from the training dataset. During tree construction, a random subset of features is considered at each split, which encourages diversity among the trees. For classification, the forest commonly predicts the class receiving the most votes. For regression, it commonly averages the predictions of individual trees. Combining diverse trees often reduces variance compared with relying on a single decision tree. Random forests can model nonlinear relationships and interactions, require relatively limited feature scaling, and provide estimates of feature importance. However, they may require substantial memory and computation, and their predictions are generally less interpretable than those of a single small tree.
Random Forest
Random forest is an ensemble learning algorithm that combines multiple decision trees to improve predictive performance and reduce the instability of individual trees.
Classification: ŷ = mode(T₁(x), T₂(x), ..., Tₘ(x)); Regression: ŷ = (1/m) Σⱼ₌₁ᵐ Tⱼ(x)
Explanation
Variables
| Symbol | Meaning |
|---|---|
Tⱼ(x) | Prediction from decision tree j |
m | Number of trees in the forest |
x | Input observation |
ŷ | Combined prediction |
Example or application
A forest containing 100 decision trees predicts whether a transaction is fraudulent. If 78 trees vote for fraud and 22 vote for legitimate, the classification output is fraud.
Text
Random forest builds many decision trees using randomized training samples and feature subsets, then combines their predictions.
Key Points
Random forest combines multiple decision trees.
It uses bootstrap samples during tree construction.
Random feature selection increases tree diversity.
Classification commonly uses majority voting.
Regression commonly uses averaging.
Ensembling can reduce prediction variance.
Applications
Fraud detection
Medical diagnosis
Credit scoring
Customer behavior prediction
Environmental forecasting
Learning Outcomes
Explain the principle of random forests.
Describe bootstrap aggregation.
Understand why random feature selection is used.
Distinguish random forest classification from regression.
Practice Questions
What is an ensemble learning algorithm?
What is bootstrap aggregation?
Why does random forest select random feature subsets?
How are predictions combined in classification and regression?
Related Concepts
Decision Trees
Ensemble Learning
Bagging
Bootstrap Sampling
Feature Importance
Common Mistakes
Assuming all trees are trained on exactly the same sample.
Confusing random forests with gradient boosting.
Assuming feature importance proves a causal relationship.
Ignoring computation and memory costs for large forests.
Sources and verification
Sources are shown as provided in the project data. Not every source has yet been independently verified. Please report errors or missing citations.
- Breiman, L. (2001). Random Forests.
- Hastie, T., Tibshirani, R., and Friedman, J. (2009). The Elements of Statistical Learning.
Suggest a correction or source · More in Artificial Intelligence & Machine Learning