A decision tree consists of a root node, internal decision nodes, branches, and leaf nodes. Each internal node applies a condition to one feature, splitting observations into groups. The process continues until a stopping condition is reached. For classification, common split criteria include Gini impurity and information gain based on entropy. For regression, splits may be selected by reducing variance or squared error. A leaf node produces the final predicted class or numerical value. Decision trees are easy to visualize and can model nonlinear relationships and feature interactions. However, deep trees can overfit training data. Limiting tree depth, setting minimum samples per leaf, and pruning can improve generalization.
Decision Trees
Decision trees are supervised learning models that make predictions through a sequence of feature-based decisions arranged in a tree structure.
Gini impurity: G = 1 - Σᵢ pᵢ²; Entropy: H = -Σᵢ pᵢ log₂(pᵢ)
Explanation
Variables
| Symbol | Meaning |
|---|---|
G | Gini impurity of a node |
H | Entropy of a node |
pᵢ | Proportion of observations belonging to class i |
i | Class index |
Example or application
A loan approval tree may first examine an applicant's income. If income exceeds a selected threshold, it may next examine credit history before producing an approval or rejection prediction.
Text
A decision tree repeatedly divides data into smaller groups using rules based on input features.
Key Points
A tree contains root, internal, branch, and leaf components.
Each decision node applies a feature-based condition.
Gini impurity and entropy are common classification split criteria.
Regression trees predict numerical values.
Deep trees can overfit.
Pruning and depth limits help control model complexity.
Applications
Credit risk assessment
Medical decision support
Customer segmentation
Fraud detection
Predictive maintenance
Learning Outcomes
Identify the main components of a decision tree.
Explain how a tree makes predictions.
Understand Gini impurity and entropy.
Describe methods for reducing decision-tree overfitting.
Practice Questions
What are the root and leaf nodes of a decision tree?
How does a decision tree divide observations?
What is Gini impurity?
Why can a deep decision tree overfit?
Related Concepts
Entropy
Gini Impurity
Information Gain
Pruning
Random Forest
Common Mistakes
Assuming a decision tree always produces the best split globally.
Allowing unlimited tree depth without validation.
Confusing classification trees with regression trees.
Interpreting feature importance as proof of causation.
Sources and verification
Sources are shown as provided in the project data. Not every source has yet been independently verified. Please report errors or missing citations.
- Breiman, L., Friedman, J. H., Olshen, R. A., and Stone, C. J. (1984). Classification and Regression Trees.
- Hastie, T., Tibshirani, R., and Friedman, J. (2009). The Elements of Statistical Learning.
Suggest a correction or source · More in Artificial Intelligence & Machine Learning