Missing values are common in real-world datasets. They may occur because of sensor failures, unanswered survey questions, data-entry errors, unavailable records, or limitations in data collection. A common first step is to measure how much data is missing and examine whether missingness follows a pattern. One strategy is deletion, which removes rows or columns containing missing values. However, deletion can reduce dataset size and introduce bias when missingness is systematic. Another strategy is imputation, which replaces missing values using methods such as the mean, median, mode, a constant value, or a model-based estimate. More advanced methods include nearest-neighbor and iterative imputation. The choice depends on the feature type, missingness mechanism, dataset size, and learning objective. Imputation parameters must be learned from training data only.
Handling Missing Values
Handling missing values involves identifying absent data and choosing appropriate strategies to manage it without introducing unnecessary bias.
Explanation
Example or application
Suppose a dataset contains the ages 20, 22, missing, 24, and 25. Mean or median imputation can estimate the missing value using the available training observations, depending on the distribution and preprocessing strategy.
Text
Missing values occur when information is unavailable, unrecorded, or intentionally omitted from a dataset.
Key Points
Missing data can occur for several different reasons.
Deletion removes incomplete observations or features.
Imputation replaces missing values with estimated values.
Mean and median imputation are common for numerical features.
Mode imputation is commonly used for categorical features.
Imputation should be fitted using training data only.
Applications
Healthcare datasets
Survey analysis
Financial records
Sensor data processing
Customer databases
Learning Outcomes
Identify common causes of missing data.
Compare deletion and imputation.
Select basic imputation strategies for numerical and categorical data.
Explain how missing-data treatment can introduce bias.
Practice Questions
What are missing values?
When might median imputation be preferable to mean imputation?
What is the difference between deletion and imputation?
Why can deleting incomplete records introduce bias?
Related Concepts
Data Cleaning
Mean Imputation
Median Imputation
Mode Imputation
Missingness Mechanisms
Common Mistakes
Replacing every missing value with zero without justification.
Assuming missing values are always random.
Ignoring how much information deletion removes.
Calculating imputation values using the test dataset.
Sources and verification
Sources are shown as provided in the project data. Not every source has yet been independently verified. Please report errors or missing citations.
- Little, R. J. A., and Rubin, D. B. (2019). Statistical Analysis with Missing Data.
- Géron, A. (2022). Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow.
Suggest a correction or source · More in Artificial Intelligence & Machine Learning