✦ScienexySCIENCE KNOWLEDGE LIBRARY
Science reference · SCI-CS-003-067

Feature Selection

Feature selection identifies a useful subset of existing input variables to improve model efficiency, interpretability, and sometimes predictive performance.

Explanation

Feature selection is the process of choosing a subset of the original features without creating entirely new representations. Filter methods evaluate features using statistical measures such as correlation, chi-square scores, or mutual information. Wrapper methods evaluate different feature subsets by repeatedly training and assessing a model. Embedded methods perform feature selection as part of model training, such as L1 regularization or tree-based feature importance. The best method depends on dataset size, feature relationships, computational cost, and model type. Feature selection should be performed within the training process or cross-validation pipeline. Selecting features using the full dataset before splitting can leak information from validation or test data and produce overly optimistic results.

Example or application

A customer-churn dataset contains 100 input columns. After analysis, a model may use only 15 relevant features, such as customer tenure, monthly charges, contract type, and support history.

Text

Selecting relevant features can reduce unnecessary complexity and help a model focus on informative input variables.

Key Points

  • Feature selection keeps a subset of existing features.

  • Filter methods use statistical criteria.

  • Wrapper methods evaluate feature subsets using a model.

  • Embedded methods select features during model training.

  • Feature selection must avoid information leakage.

Applications

  • Customer churn prediction

  • Medical diagnosis

  • Financial risk modeling

  • Text classification

  • High-dimensional scientific datasets

Learning Outcomes

  • Define feature selection.

  • Compare filter, wrapper, and embedded methods.

  • Explain why reducing irrelevant features can help.

  • Recognize data leakage risks in feature selection.

Practice Questions

  • What is feature selection?

  • How do filter and wrapper methods differ?

  • What is an example of an embedded feature-selection method?

  • Why should feature selection be performed inside cross-validation?

Related Concepts

  • Feature Importance

  • Mutual Information

  • Regularization

  • Dimensionality Reduction

  • Cross-Validation

Common Mistakes

  • Assuming every low-correlation feature is useless.

  • Ignoring interactions between features.

  • Selecting features using the complete dataset before splitting.

  • Confusing feature selection with feature extraction.

Sources and verification

Sources are shown as provided in the project data. Not every source has yet been independently verified. Please report errors or missing citations.

  • Kuhn, M., and Johnson, K. (2019). Feature Engineering and Selection: A Practical Approach for Predictive Models.
  • Guyon, I., and Elisseeff, A. (2003). An Introduction to Variable and Feature Selection.

Suggest a correction or source · More in Artificial Intelligence & Machine Learning