Cross-validation is used to estimate model generalization performance and support model selection when the available dataset is limited. Instead of relying on a single training-validation split, the method evaluates a model across multiple data partitions.
In k-fold cross-validation, the dataset is divided into k approximately equal folds. The model is trained k times. During each run, one fold is held out for validation, while the remaining k − 1 folds are used for training. Each fold serves as the validation set exactly once. The validation scores are then averaged to produce an overall estimate.
Other techniques are useful for particular data structures. Stratified k-fold cross-validation attempts to preserve class proportions in each fold and is useful for classification with imbalanced classes. Leave-one-out cross-validation uses one observation for validation in each run and all remaining observations for training. Time-series cross-validation respects chronological order, training on earlier observations and evaluating on later ones.
Cross-validation is commonly used for comparing algorithms and selecting hyperparameters. However, it must be applied carefully. Data leakage can occur if information from validation folds influences preprocessing or model training. Preprocessing steps such as feature scaling and feature selection should be fitted separately within each training fold, often through a pipeline.
When substantial model selection is performed, nested cross-validation can provide a less biased evaluation procedure by separating inner model selection from outer performance estimation. A final untouched test set may also be retained for independent evaluation.
Cross-Validation Techniques
Cross-validation is a model evaluation technique that repeatedly divides available data into training and validation portions to estimate how well a model may perform on unseen observations.
CV Score = (1/k) Σⱼ₌₁ᵏ Sⱼ
Explanation
Variables
| Symbol | Meaning |
|---|---|
k | Number of folds |
Sⱼ | Validation score obtained from fold j |
CV Score | Average cross-validation score |
Example or application
A dataset contains 1,000 customer records. In five-fold cross-validation, it is divided into five groups of approximately 200 records. The model is trained five times, each time using approximately 800 records for training and a different group of 200 for validation. If the five validation accuracies are 88%, 90%, 87%, 91%, and 89%, the average cross-validation accuracy is 89%.
Text
Cross-validation estimates how well a machine learning model may generalize by evaluating it across multiple data splits. In k-fold cross-validation, data is divided into k folds, and the model is trained k times, using a different fold for validation each time. The resulting scores are combined, commonly by taking their average. Stratified and time-series cross-validation address special data structures. Preprocessing must be performed within each training fold to prevent data leakage.
Key Points
Cross-validation evaluates models across multiple data partitions.
K-fold cross-validation uses each fold for validation once.
Stratified folds help preserve class proportions.
Leave-one-out uses one observation as the validation set per run.
Time-series validation must respect chronological order.
Preprocessing should be fitted independently within each training fold.
Nested cross-validation separates hyperparameter selection from performance estimation.
Applications
Comparing machine learning algorithms
Selecting model hyperparameters
Evaluating models on small datasets
Estimating classification and regression performance
Detecting unstable model performance across data splits
Learning Outcomes
Explain the purpose of cross-validation.
Describe the steps of k-fold cross-validation.
Distinguish stratified, leave-one-out, and time-series validation.
Explain how data leakage can affect cross-validation results.
Practice Questions
How does five-fold cross-validation work?
Why is stratified cross-validation useful for imbalanced classification?
Why should preprocessing be fitted separately within each training fold?
Related Concepts
Training and Validation Sets
Hyperparameter Tuning
Data Leakage
Model Selection
Nested Cross-Validation
Common Mistakes
Allowing validation information to influence training preprocessing.
Randomly shuffling time-series observations when chronological order matters.
Treating cross-validation performance as a completely independent final test result.
Sources and verification
Sources are shown as provided in the project data. Not every source has yet been independently verified. Please report errors or missing citations.
- Hastie, T., Tibshirani, R., and Friedman, J. (2009). The Elements of Statistical Learning. Springer.
- Kohavi, R. (1995). A Study of Cross-Validation and Bootstrap for Accuracy Estimation and Model Selection. Proceedings of the 14th International Joint Conference on Artificial Intelligence.
Suggest a correction or source · More in Artificial Intelligence & Machine Learning