✦ScienexySCIENCE KNOWLEDGE LIBRARY
Science reference · SCI-CS-003-076

Decision Trees

Decision trees are supervised learning models that make predictions through a sequence of feature-based decisions arranged in a tree structure.

Gini impurity: G = 1 - Σᵢ pᵢ²; Entropy: H = -Σᵢ pᵢ log₂(pᵢ)

Explanation

A decision tree consists of a root node, internal decision nodes, branches, and leaf nodes. Each internal node applies a condition to one feature, splitting observations into groups. The process continues until a stopping condition is reached. For classification, common split criteria include Gini impurity and information gain based on entropy. For regression, splits may be selected by reducing variance or squared error. A leaf node produces the final predicted class or numerical value. Decision trees are easy to visualize and can model nonlinear relationships and feature interactions. However, deep trees can overfit training data. Limiting tree depth, setting minimum samples per leaf, and pruning can improve generalization.

Variables

SymbolMeaning

G

Gini impurity of a node

H

Entropy of a node

pᵢ

Proportion of observations belonging to class i

i

Class index

Example or application

A loan approval tree may first examine an applicant's income. If income exceeds a selected threshold, it may next examine credit history before producing an approval or rejection prediction.

Text

A decision tree repeatedly divides data into smaller groups using rules based on input features.

Key Points

  • A tree contains root, internal, branch, and leaf components.

  • Each decision node applies a feature-based condition.

  • Gini impurity and entropy are common classification split criteria.

  • Regression trees predict numerical values.

  • Deep trees can overfit.

  • Pruning and depth limits help control model complexity.

Applications

  • Credit risk assessment

  • Medical decision support

  • Customer segmentation

  • Fraud detection

  • Predictive maintenance

Learning Outcomes

  • Identify the main components of a decision tree.

  • Explain how a tree makes predictions.

  • Understand Gini impurity and entropy.

  • Describe methods for reducing decision-tree overfitting.

Practice Questions

  • What are the root and leaf nodes of a decision tree?

  • How does a decision tree divide observations?

  • What is Gini impurity?

  • Why can a deep decision tree overfit?

Related Concepts

  • Entropy

  • Gini Impurity

  • Information Gain

  • Pruning

  • Random Forest

Common Mistakes

  • Assuming a decision tree always produces the best split globally.

  • Allowing unlimited tree depth without validation.

  • Confusing classification trees with regression trees.

  • Interpreting feature importance as proof of causation.

Sources and verification

Sources are shown as provided in the project data. Not every source has yet been independently verified. Please report errors or missing citations.

  • Breiman, L., Friedman, J. H., Olshen, R. A., and Stone, C. J. (1984). Classification and Regression Trees.
  • Hastie, T., Tibshirani, R., and Friedman, J. (2009). The Elements of Statistical Learning.

Suggest a correction or source · More in Artificial Intelligence & Machine Learning