Overview
Scientific computing can produce complex results that are difficult to evaluate by inspection alone. A simulation may execute without errors and generate realistic-looking visualizations while still containing programming mistakes, numerical instability, incorrect assumptions, or inappropriate physical models. Verification, validation, and reproducibility provide complementary ways to evaluate computational research. They help establish whether the implementation is correct, whether the model is scientifically adequate, and whether the results can be independently examined or regenerated.
Verification
Definition
Verification is the process of determining whether a computational implementation correctly represents and solves the mathematical or computational model that was intended.
Central Question
Are we solving the mathematical problem correctly?
Focus Areas
Correctness of equations and algorithms.
Implementation of mathematical relationships.
Software logic and data handling.
Numerical stability and convergence.
Correct use of initial and boundary conditions.
Consistency with mathematical properties and known solutions.
Methods
| Method | Explanation |
|---|---|
Code review | Inspect the implementation to identify logical errors, incorrect equations, unsafe assumptions, and poor numerical practices. |
Unit testing | Test individual functions or components using controlled inputs and expected outputs. |
Integration testing | Check whether different software components work correctly when combined. |
Analytical comparison | Compare numerical results with an exact analytical solution when one is available. |
Benchmark testing | Compare results with trusted reference calculations or established benchmark problems. |
Convergence testing | Repeat numerical calculations at different resolutions or step sizes to investigate whether the results approach a stable solution. |
Conservation checks | Check whether quantities that should be conserved, such as mass or energy in a suitable model, remain consistent within expected numerical error. |
Validation
Definition
Validation is the process of evaluating whether a mathematical and computational model represents the real-world system sufficiently well for its intended application.
Central Question
Are we solving the right scientific problem with a sufficiently appropriate model?
Focus Areas
Physical or biological assumptions.
Representation of relevant system mechanisms.
Agreement with experimental observations.
Predictive performance under independent conditions.
Applicability within the intended range of conditions.
Adequacy of model structure and parameters.
Methods
| Method | Explanation |
|---|---|
Experimental comparison | Compare model predictions with observations from controlled experiments. |
Observational comparison | Compare predictions with independently collected real-world measurements. |
Independent dataset testing | Evaluate model performance using data not used to calibrate the model. |
Sensitivity analysis | Determine whether model outputs change excessively under plausible variations in parameters or assumptions. |
Physical consistency checks | Evaluate whether predictions obey known physical constraints and scientific principles. |
Uncertainty comparison | Consider whether model uncertainty is consistent with measurement uncertainty and the variability of the real system. |
Reproducibility
Definition
Reproducibility in computational research concerns the ability to independently regenerate or check reported computational results using the documented procedures, data, code, and computational conditions.
Central Question
Can another researcher inspect and repeat the computational investigation sufficiently well to obtain comparable results?
Required Information
Source code and relevant scripts.
Original or appropriately accessible input data.
Data preprocessing and transformation procedures.
Model equations and assumptions.
Software and library versions.
Parameter values and simulation settings.
Random seeds and random-number-generation details when relevant.
Hardware or computational environment details when they materially affect results.
Documentation explaining how to execute the workflow.
Information about data access restrictions and licensing.
Practices
Use version control to preserve changes in code.
Record software dependencies and versions.
Provide clear installation and execution instructions.
Document data sources and transformations.
Preserve scripts, configuration files, and relevant metadata.
Use containers or environment-management tools when useful.
Save random seeds for stochastic experiments where appropriate.
Archive important versions of data and code.
Clearly state any steps that cannot be shared or reproduced.
Comparison
| Dimension | Verification | Validation | Reproducibility |
|---|---|---|---|
Main purpose | Check correctness of computational implementation. | Evaluate adequacy of the model for the real system. | Enable independent regeneration or examination of results. |
Central concern | Software, algorithms, equations, and numerical implementation. | Scientific assumptions, real-world behavior, and empirical evidence. | Data, code, dependencies, parameters, documentation, and execution procedures. |
Typical evidence | Tests, analytical solutions, benchmarks, and convergence studies. | Experimental observations, independent datasets, and scientific evidence. | Archived code, accessible data, environment descriptions, and execution records. |
Possible failure | The program implements the wrong equation or algorithm. | The model uses unrealistic assumptions or fails to represent important real-world behavior. | Another researcher cannot reconstruct the reported computational procedure. |
Numerical Quality Concepts
Accuracy
How close a calculated result is to a reference or true value.
Precision
The resolution or consistency of numerical representation or measurement.
Stability
Whether numerical errors remain controlled during computation.
Convergence
Whether numerical approximations approach a limiting solution as the numerical resolution is refined.
Consistency
Whether a numerical method appropriately approximates the underlying mathematical problem as the discretization is refined.
Uncertainty
A quantified or characterized lack of certainty associated with measurements, parameters, model structure, or numerical results.
Formula
| Name | Equation | Meaning | Variables | Limitation |
|---|---|---|---|---|
Absolute numerical error | E_abs = |x_num - x_ref| | Measures the magnitude of the difference between a numerical result and a reference value. | E abs Absolute error in the same unit as the quantity. X num Numerical result. X ref Reference or exact value. | |
Root Mean Square Error | RMSE = sqrt((1/n) Σ(i=1 to n) (y_i - ŷ_i)^2) | Measures the typical magnitude of prediction errors using squared residuals. It is commonly used when comparing model predictions with observed values. | RMSE Root Mean Square Error, expressed in the same unit as the measured output. N Number of observations. Y i Observed value for observation i. Ŷ i Predicted value for observation i. I Observation index. | RMSE emphasizes larger errors because residuals are squared. A low RMSE alone does not prove that a model is scientifically valid. |
Verification And Validation In Practice
Verification Example
For a program that solves a simple projectile-motion equation, compare its calculated position with the known analytical expression at selected times and test limiting cases such as zero initial velocity.
Validation Example
Compare predicted projectile trajectories with measurements from a real experiment while accounting for air resistance, measurement uncertainty, and the assumptions of the model.
Reproducibility Example
Provide the source code, input values, software environment, execution instructions, and data needed for another researcher to regenerate the trajectory graph.
Worked Example
Project
A numerical program simulates heat diffusion through a metal plate.
Verification
Check that the implemented heat equation matches the intended mathematical formulation.
Test the numerical solver using a simplified problem with a known analytical solution.
Check whether temperature values remain physically reasonable.
Repeat the simulation with smaller spatial and temporal steps.
Check whether the numerical solution approaches a stable result as resolution increases.
Validation
Measure temperatures at selected locations in a physical experiment.
Compare the model predictions with the independent measurements.
Investigate differences caused by material properties, boundary conditions, heat losses, or measurement uncertainty.
Evaluate whether the model is adequate for the intended prediction task.
Reproducibility
Store the source code in a version-controlled repository.
Record material properties, initial conditions, and boundary conditions.
Document the numerical method and resolution.
Record software dependencies and execution instructions.
Preserve the dataset and analysis script.
Interpretation
Verification provides evidence that the numerical program solves the intended mathematical problem. Validation provides evidence about whether the mathematical model represents the real heat-transfer process. Reproducibility allows others to examine and repeat the investigation.
Uncertainty And Limitations
Verification cannot prove that the scientific model is realistic; it examines computational correctness.
Validation is limited by the quality, quantity, and range of available experimental evidence.
Reproducing the same result does not automatically establish that the result is scientifically correct.
Some stochastic simulations may produce different outputs unless random seeds and sampling procedures are controlled.
Hardware differences and numerical libraries may lead to small floating-point differences.
Some datasets cannot be publicly shared because of privacy, security, ownership, or licensing restrictions.
Common Mistakes
Assuming that a program is verified simply because it runs without crashing.
Confusing verification with validation.
Using the same data for both model calibration and independent validation without acknowledging the limitation.
Ignoring numerical convergence and stability.
Reporting results without documenting software versions and parameters.
Assuming that reproducibility guarantees scientific validity.
Ignoring uncertainty in experimental measurements and model parameters.
Practical Learning
Write unit tests for a program that calculates a known mathematical result.
Compare a numerical integration method with an analytical solution.
Repeat a simulation using different numerical step sizes and investigate convergence.
Compare a model prediction with a small independently collected dataset.
Place the program and its documentation in a version-controlled repository.
Recreate the computational result using a clean software environment and documented instructions.
Learning Objectives
Define verification, validation, and reproducibility.
Explain the different questions answered by these three quality principles.
Identify suitable verification and validation techniques.
Understand numerical accuracy, stability, convergence, and uncertainty.
Design a reproducible computational experiment.
Recognize why a computational result must be evaluated before scientific conclusions are drawn.
Assessment Questions
What is the difference between verification and validation?
Why does a successful software execution not prove that a scientific model is correct?
How can an analytical solution be used for code verification?
Why should independent data be used for model validation?
What information should be preserved to reproduce a scientific simulation?
Can a reproducible computational result still be scientifically wrong? Explain.