A scientific model must be evaluated against evidence to determine whether it is suitable for its intended purpose. Testing involves examining the consequences of a model through experiments, observations, simulations or comparisons with known results. Validation assesses how well the model represents the real system for a specified application. Evaluation is broader and considers the model's assumptions, structure, accuracy, reliability, uncertainty, explanatory ability and limitations. Researchers may compare predicted values with measured data, test the model using independent datasets, investigate its behavior under different conditions and examine whether its predictions remain consistent across relevant situations. Calibration and validation are related but distinct activities. Calibration adjusts selected model parameters using observations, whereas validation examines model performance using evidence that is not simply the same data used for parameter adjustment. Verification is also distinct: it examines whether the mathematical or computational model has been implemented and solved correctly. A model can be correctly implemented but still represent reality poorly. Similarly, a model may fit existing observations but perform poorly when applied to new conditions. Reliable evaluation therefore requires clear objectives, appropriate evidence, transparent methods and careful interpretation of uncertainty.
Testing, Validation and Evaluation of Scientific Models
Testing, validation and evaluation are essential processes used to determine whether a scientific model adequately represents a system and produces reliable results for its intended purpose. They involve comparing predictions with observations, examining assumptions, assessing uncertainty and evaluating performance under relevant conditions.
No single formula can validate every scientific model. Several quantitative measures may be used to evaluate model predictions.
Mean Absolute Error (MAE):
MAE = (1/n) Σᵢ₌₁ⁿ |yᵢ − ŷᵢ|
Root Mean Square Error (RMSE):
RMSE = √[(1/n) Σᵢ₌₁ⁿ (yᵢ − ŷᵢ)²]
Coefficient of Determination:
R² = 1 − [Σᵢ₌₁ⁿ (yᵢ − ŷᵢ)² / Σᵢ₌₁ⁿ (yᵢ − ȳ)²]
These measures are useful in appropriate contexts, but no individual metric establishes universal scientific validity.
Explanation
Variables
| Symbol | Meaning | Si Unit |
|---|---|---|
n | Number of observations or data points | Dimensionless |
yᵢ | Observed value for the i-th data point | Depends on the measured quantity |
ŷᵢ | Model-predicted value for the i-th data point | Same as the observed quantity |
ȳ | Mean of the observed values | Same as the observed quantity |
MAE | Mean absolute prediction error | Same as the predicted quantity |
RMSE | Root mean square prediction error | Same as the predicted quantity |
R² | Coefficient of determination | Dimensionless |
Example or application
A worked example has not yet been added.
Definition
Testing, validation and evaluation of scientific models are systematic procedures for examining a model's implementation, agreement with empirical evidence, predictive performance, assumptions, uncertainty and suitability for a defined scientific purpose.
Purpose
To establish whether a model is implemented correctly, supported by evidence and sufficiently reliable for its intended explanatory or predictive application.
Importance
Model evaluation prevents unsupported confidence in scientific representations. It helps researchers identify errors, detect unsuitable assumptions, quantify prediction performance and determine when a model needs improvement or replacement.
Key Concepts
| Concept | Description |
|---|---|
Model testing | Examining a model through experiments, observations, simulations or logical checks to determine whether its consequences are consistent with evidence. |
Model verification | Determining whether the mathematical formulation or computational implementation has been carried out correctly. It focuses on whether the model is solved or implemented as intended. |
Model validation | Determining whether the model represents the real-world system adequately for a specified purpose and domain of application. |
Model calibration | Estimating or adjusting model parameters using observed data to improve agreement between model outputs and measurements. |
Model evaluation | A broader assessment of a model's performance, assumptions, uncertainty, explanatory value, limitations and suitability. |
Predictive evaluation | Assessing how accurately a model predicts observations that were not used to construct or calibrate it. |
Testing Methods
| Method | Description | Use |
|---|---|---|
Experimental testing | Compare model predictions with controlled experimental measurements. | Testing physical, chemical and biological models under controlled conditions. |
Observational comparison | Compare model outputs with naturally occurring phenomena and collected observational datasets. | Evaluating climate, ecological, astronomical and geological models. |
Independent data validation | Evaluate model performance using a dataset separate from the data used for model development or calibration. | Assessing whether a model generalizes beyond its original observations. |
Sensitivity analysis | Examine how variations in parameters and assumptions affect model outputs. | Identifying influential variables and uncertain assumptions. |
Cross-validation | Repeatedly divide available data into training and evaluation subsets to estimate predictive performance. | Evaluating statistical and data-driven models. |
Comparison with established models | Compare a model with previously tested theoretical or empirical representations. | Determining whether a new model offers improved explanatory or predictive performance. |
Evaluation Metrics
| Metric | Description | Interpretation |
|---|---|---|
Mean Absolute Error | Measures the average magnitude of prediction errors without considering their direction. | Lower values generally indicate smaller average prediction errors. |
Root Mean Square Error | Measures prediction error while giving greater weight to larger errors. | Lower values generally indicate better numerical agreement, although the metric is sensitive to outliers. |
Coefficient of Determination | Measures how much of the observed variation is accounted for by a fitted statistical model in the context where R² is appropriate. | A higher value may indicate a stronger fit, but does not independently establish causation or scientific validity. |
Residual analysis | Examines the differences between observed and predicted values to identify systematic patterns or model inadequacies. | Patterns in residuals may indicate missing relationships, unsuitable assumptions or unmodeled effects. |
Worked Example
Title
Evaluating a Simple Motion Model
Problem
A model predicts the position of an object at three different times. Its predictions are compared with measured positions.
Observations
| Time | Observed Position | Predicted Position | Absolute Error |
|---|---|---|---|
1 s | 5 m | 4.8 m | 0.2 m |
2 s | 9 m | 9.2 m | 0.2 m |
3 s | 13 m | 12.7 m | 0.3 m |
Calculation
MAE = (0.2 + 0.2 + 0.3) / 3 = 0.233 m approximately.
Interpretation
The model's average absolute prediction error for these observations is approximately 0.233 m.
Limitation
Three observations alone are insufficient to establish that the model is valid for every time, speed or experimental condition.
Validation Workflow
Define the intended purpose and application domain of the model.
Specify measurable criteria for acceptable performance.
Check the mathematical formulation and computational implementation.
Calibrate parameters where appropriate using suitable observations.
Evaluate predictions against independent experimental or observational evidence.
Examine residuals, uncertainty and systematic errors.
Test the model under different relevant conditions.
Document the evidence, limitations and remaining uncertainties.
Revise the model when its performance is inadequate.
Repeat evaluation after significant changes to the model.
Challenges
Experimental measurements may contain noise and systematic errors.
Available data may not cover the full range of conditions in which the model will be used.
A model may fit existing data but fail to generalize to new observations.
Some scientific systems cannot be reproduced easily under controlled experimental conditions.
Different evaluation metrics may emphasize different aspects of model performance.
Complex models may be difficult to validate because many interacting mechanisms contribute to their outputs.
Limitations
Validation is always related to a specified purpose and application domain.
Agreement with observations does not prove that a model is the only possible explanation.
A high statistical fit does not automatically establish causal relationships.
A model validated under one set of conditions may not remain reliable under substantially different conditions.
Insufficient or biased data can produce misleading evaluation results.
Model evaluation cannot remove every form of uncertainty.
Common Misconceptions
Verification and validation are not interchangeable concepts.
Calibration does not automatically demonstrate that a model is valid.
A model that matches historical data is not necessarily reliable for future predictions.
A high R² value alone does not prove scientific correctness.
No scientific model is universally valid for every purpose and condition.
Applications
Physics tests theoretical models through controlled experiments and precise measurements.
Chemistry evaluates molecular and reaction models against experimental observations.
Biology validates physiological, genetic and ecological models using laboratory and field evidence.
Climate science evaluates model projections against historical observations and independent datasets.
Astronomy compares predicted celestial behavior with telescope observations.
Engineering evaluates structural and computational models through simulations and physical testing.
Medicine evaluates statistical and physiological models using appropriate clinical and biological evidence.
Reference
National Institute of Standards and Technology, NIST Technical Note 1297: Guidelines for Evaluating and Expressing the Uncertainty of NIST Measurement Results.
National Academies of Sciences, Engineering, and Medicine, Models in Environmental Regulatory Decision Making.
ISO 9000, Quality Management Systems — Fundamentals and Vocabulary.
Related Topic
SCI-REF-005-010
SCI-REF-005-011
SCI-REF-005-012
SCI-REF-005-014
SCI-REF-005-015
Sources and verification
Sources are shown as provided in the project data. Not every source has yet been independently verified. Please report errors or missing citations.
- Source verification pending; check an authoritative textbook, university, standards body, or peer-reviewed source.