✦ScienexySCIENCE KNOWLEDGE LIBRARY
Science reference · SCI-CS-003-095

Cross-Validation Techniques

Cross-validation is a model evaluation technique that repeatedly divides available data into training and validation portions to estimate how well a model may perform on unseen observations.

CV Score = (1/k) Σⱼ₌₁ᵏ Sⱼ

Explanation

Cross-validation is used to estimate model generalization performance and support model selection when the available dataset is limited. Instead of relying on a single training-validation split, the method evaluates a model across multiple data partitions.

In k-fold cross-validation, the dataset is divided into k approximately equal folds. The model is trained k times. During each run, one fold is held out for validation, while the remaining k − 1 folds are used for training. Each fold serves as the validation set exactly once. The validation scores are then averaged to produce an overall estimate.

Other techniques are useful for particular data structures. Stratified k-fold cross-validation attempts to preserve class proportions in each fold and is useful for classification with imbalanced classes. Leave-one-out cross-validation uses one observation for validation in each run and all remaining observations for training. Time-series cross-validation respects chronological order, training on earlier observations and evaluating on later ones.

Cross-validation is commonly used for comparing algorithms and selecting hyperparameters. However, it must be applied carefully. Data leakage can occur if information from validation folds influences preprocessing or model training. Preprocessing steps such as feature scaling and feature selection should be fitted separately within each training fold, often through a pipeline.

When substantial model selection is performed, nested cross-validation can provide a less biased evaluation procedure by separating inner model selection from outer performance estimation. A final untouched test set may also be retained for independent evaluation.

Variables

SymbolMeaning

k

Number of folds

Sⱼ

Validation score obtained from fold j

CV Score

Average cross-validation score

Example or application

A dataset contains 1,000 customer records. In five-fold cross-validation, it is divided into five groups of approximately 200 records. The model is trained five times, each time using approximately 800 records for training and a different group of 200 for validation. If the five validation accuracies are 88%, 90%, 87%, 91%, and 89%, the average cross-validation accuracy is 89%.

Text

Cross-validation estimates how well a machine learning model may generalize by evaluating it across multiple data splits. In k-fold cross-validation, data is divided into k folds, and the model is trained k times, using a different fold for validation each time. The resulting scores are combined, commonly by taking their average. Stratified and time-series cross-validation address special data structures. Preprocessing must be performed within each training fold to prevent data leakage.

Key Points

  • Cross-validation evaluates models across multiple data partitions.

  • K-fold cross-validation uses each fold for validation once.

  • Stratified folds help preserve class proportions.

  • Leave-one-out uses one observation as the validation set per run.

  • Time-series validation must respect chronological order.

  • Preprocessing should be fitted independently within each training fold.

  • Nested cross-validation separates hyperparameter selection from performance estimation.

Applications

  • Comparing machine learning algorithms

  • Selecting model hyperparameters

  • Evaluating models on small datasets

  • Estimating classification and regression performance

  • Detecting unstable model performance across data splits

Learning Outcomes

  • Explain the purpose of cross-validation.

  • Describe the steps of k-fold cross-validation.

  • Distinguish stratified, leave-one-out, and time-series validation.

  • Explain how data leakage can affect cross-validation results.

Practice Questions

  • How does five-fold cross-validation work?

  • Why is stratified cross-validation useful for imbalanced classification?

  • Why should preprocessing be fitted separately within each training fold?

Related Concepts

  • Training and Validation Sets

  • Hyperparameter Tuning

  • Data Leakage

  • Model Selection

  • Nested Cross-Validation

Common Mistakes

  • Allowing validation information to influence training preprocessing.

  • Randomly shuffling time-series observations when chronological order matters.

  • Treating cross-validation performance as a completely independent final test result.

Sources and verification

Sources are shown as provided in the project data. Not every source has yet been independently verified. Please report errors or missing citations.

  • Hastie, T., Tibshirani, R., and Friedman, J. (2009). The Elements of Statistical Learning. Springer.
  • Kohavi, R. (1995). A Study of Cross-Validation and Bootstrap for Accuracy Estimation and Model Selection. Proceedings of the 14th International Joint Conference on Artificial Intelligence.

Suggest a correction or source · More in Artificial Intelligence & Machine Learning