✦ScienexySCIENCE KNOWLEDGE LIBRARY
Science reference · SCI-CS-003-063

Data Cleaning and Preprocessing

Data cleaning and preprocessing transform raw data into a consistent and usable format before machine learning models are trained.

Explanation

Data cleaning focuses on identifying and correcting problems such as duplicate records, invalid values, inconsistent labels, incorrect data types, and formatting errors. Data preprocessing is broader and includes preparing data for machine learning algorithms. Common operations include handling missing values, encoding categorical variables, scaling numerical features, parsing dates, removing irrelevant fields, and transforming text or images into suitable representations. These operations should be performed carefully to preserve meaningful information. Preprocessing steps that learn parameters from data, such as calculating scaling statistics or selecting features, must be fitted using training data only. Applying those fitted transformations to validation and test data prevents information leakage.

Example or application

A customer dataset may contain duplicate customer records, inconsistent country names, missing ages, and dates written in different formats. Cleaning and preprocessing make these records consistent before model training.

Text

Raw datasets often contain errors, inconsistent formats, duplicates, and irrelevant information that must be addressed.

Key Points

  • Data cleaning corrects errors and inconsistencies.

  • Preprocessing prepares data for model requirements.

  • Duplicate and invalid records should be investigated.

  • Transformations must preserve useful information.

  • Preprocessing must avoid data leakage.

Applications

  • Customer analytics

  • Medical prediction

  • Financial modeling

  • Text classification

  • Image recognition

Learning Outcomes

  • Differentiate data cleaning from data preprocessing.

  • Identify common problems in raw datasets.

  • Describe typical preprocessing operations.

  • Explain how preprocessing can introduce data leakage.

Practice Questions

  • What is the difference between data cleaning and preprocessing?

  • Why should duplicate records be investigated?

  • What are common preprocessing operations?

  • Why should preprocessing transformations be fitted only on training data?

Related Concepts

  • Missing Values

  • Data Transformation

  • Feature Scaling

  • Categorical Encoding

  • Data Leakage

Common Mistakes

  • Deleting records without understanding why values are missing.

  • Applying transformations inconsistently across datasets.

  • Using test data to calculate preprocessing parameters.

  • Removing unusual values that may contain important information.

Sources and verification

Sources are shown as provided in the project data. Not every source has yet been independently verified. Please report errors or missing citations.

  • Géron, A. (2022). Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow.
  • Kuhn, M., and Johnson, K. (2019). Feature Engineering and Selection: A Practical Approach for Predictive Models.

Suggest a correction or source · More in Artificial Intelligence & Machine Learning