✦ScienexySCIENCE KNOWLEDGE LIBRARY
topic · SCI-CS-005-009

Verification, Validation and Reproducibility

Verification, Validation and Reproducibility are essential quality principles in scientific computing. Verification examines whether software and numerical procedures correctly implement the intended mathematical model. Validation evaluates whether the model adequately represents the real system for its intended purpose. Reproducibility concerns whether computational results can be independently regenerated or checked using documented data, code, parameters, and computational environments.

Explanation

Overview

Scientific computing can produce complex results that are difficult to evaluate by inspection alone. A simulation may execute without errors and generate realistic-looking visualizations while still containing programming mistakes, numerical instability, incorrect assumptions, or inappropriate physical models. Verification, validation, and reproducibility provide complementary ways to evaluate computational research. They help establish whether the implementation is correct, whether the model is scientifically adequate, and whether the results can be independently examined or regenerated.

Verification

Definition

Verification is the process of determining whether a computational implementation correctly represents and solves the mathematical or computational model that was intended.

Central Question

Are we solving the mathematical problem correctly?

Focus Areas

  • Correctness of equations and algorithms.

  • Implementation of mathematical relationships.

  • Software logic and data handling.

  • Numerical stability and convergence.

  • Correct use of initial and boundary conditions.

  • Consistency with mathematical properties and known solutions.

Methods

MethodExplanation

Code review

Inspect the implementation to identify logical errors, incorrect equations, unsafe assumptions, and poor numerical practices.

Unit testing

Test individual functions or components using controlled inputs and expected outputs.

Integration testing

Check whether different software components work correctly when combined.

Analytical comparison

Compare numerical results with an exact analytical solution when one is available.

Benchmark testing

Compare results with trusted reference calculations or established benchmark problems.

Convergence testing

Repeat numerical calculations at different resolutions or step sizes to investigate whether the results approach a stable solution.

Conservation checks

Check whether quantities that should be conserved, such as mass or energy in a suitable model, remain consistent within expected numerical error.

Validation

Definition

Validation is the process of evaluating whether a mathematical and computational model represents the real-world system sufficiently well for its intended application.

Central Question

Are we solving the right scientific problem with a sufficiently appropriate model?

Focus Areas

  • Physical or biological assumptions.

  • Representation of relevant system mechanisms.

  • Agreement with experimental observations.

  • Predictive performance under independent conditions.

  • Applicability within the intended range of conditions.

  • Adequacy of model structure and parameters.

Methods

MethodExplanation

Experimental comparison

Compare model predictions with observations from controlled experiments.

Observational comparison

Compare predictions with independently collected real-world measurements.

Independent dataset testing

Evaluate model performance using data not used to calibrate the model.

Sensitivity analysis

Determine whether model outputs change excessively under plausible variations in parameters or assumptions.

Physical consistency checks

Evaluate whether predictions obey known physical constraints and scientific principles.

Uncertainty comparison

Consider whether model uncertainty is consistent with measurement uncertainty and the variability of the real system.

Reproducibility

Definition

Reproducibility in computational research concerns the ability to independently regenerate or check reported computational results using the documented procedures, data, code, and computational conditions.

Central Question

Can another researcher inspect and repeat the computational investigation sufficiently well to obtain comparable results?

Required Information

  • Source code and relevant scripts.

  • Original or appropriately accessible input data.

  • Data preprocessing and transformation procedures.

  • Model equations and assumptions.

  • Software and library versions.

  • Parameter values and simulation settings.

  • Random seeds and random-number-generation details when relevant.

  • Hardware or computational environment details when they materially affect results.

  • Documentation explaining how to execute the workflow.

  • Information about data access restrictions and licensing.

Practices

  • Use version control to preserve changes in code.

  • Record software dependencies and versions.

  • Provide clear installation and execution instructions.

  • Document data sources and transformations.

  • Preserve scripts, configuration files, and relevant metadata.

  • Use containers or environment-management tools when useful.

  • Save random seeds for stochastic experiments where appropriate.

  • Archive important versions of data and code.

  • Clearly state any steps that cannot be shared or reproduced.

Comparison

DimensionVerificationValidationReproducibility

Main purpose

Check correctness of computational implementation.

Evaluate adequacy of the model for the real system.

Enable independent regeneration or examination of results.

Central concern

Software, algorithms, equations, and numerical implementation.

Scientific assumptions, real-world behavior, and empirical evidence.

Data, code, dependencies, parameters, documentation, and execution procedures.

Typical evidence

Tests, analytical solutions, benchmarks, and convergence studies.

Experimental observations, independent datasets, and scientific evidence.

Archived code, accessible data, environment descriptions, and execution records.

Possible failure

The program implements the wrong equation or algorithm.

The model uses unrealistic assumptions or fails to represent important real-world behavior.

Another researcher cannot reconstruct the reported computational procedure.

Numerical Quality Concepts

Accuracy

How close a calculated result is to a reference or true value.

Precision

The resolution or consistency of numerical representation or measurement.

Stability

Whether numerical errors remain controlled during computation.

Convergence

Whether numerical approximations approach a limiting solution as the numerical resolution is refined.

Consistency

Whether a numerical method appropriately approximates the underlying mathematical problem as the discretization is refined.

Uncertainty

A quantified or characterized lack of certainty associated with measurements, parameters, model structure, or numerical results.

Formula

NameEquationMeaningVariablesLimitation

Absolute numerical error

E_abs = |x_num - x_ref|

Measures the magnitude of the difference between a numerical result and a reference value.

E abs

Absolute error in the same unit as the quantity.

X num

Numerical result.

X ref

Reference or exact value.

Root Mean Square Error

RMSE = sqrt((1/n) Σ(i=1 to n) (y_i - ŷ_i)^2)

Measures the typical magnitude of prediction errors using squared residuals. It is commonly used when comparing model predictions with observed values.

RMSE

Root Mean Square Error, expressed in the same unit as the measured output.

N

Number of observations.

Y i

Observed value for observation i.

Ŷ i

Predicted value for observation i.

I

Observation index.

RMSE emphasizes larger errors because residuals are squared. A low RMSE alone does not prove that a model is scientifically valid.

Verification And Validation In Practice

Verification Example

For a program that solves a simple projectile-motion equation, compare its calculated position with the known analytical expression at selected times and test limiting cases such as zero initial velocity.

Validation Example

Compare predicted projectile trajectories with measurements from a real experiment while accounting for air resistance, measurement uncertainty, and the assumptions of the model.

Reproducibility Example

Provide the source code, input values, software environment, execution instructions, and data needed for another researcher to regenerate the trajectory graph.

Worked Example

Project

A numerical program simulates heat diffusion through a metal plate.

Verification

  • Check that the implemented heat equation matches the intended mathematical formulation.

  • Test the numerical solver using a simplified problem with a known analytical solution.

  • Check whether temperature values remain physically reasonable.

  • Repeat the simulation with smaller spatial and temporal steps.

  • Check whether the numerical solution approaches a stable result as resolution increases.

Validation

  • Measure temperatures at selected locations in a physical experiment.

  • Compare the model predictions with the independent measurements.

  • Investigate differences caused by material properties, boundary conditions, heat losses, or measurement uncertainty.

  • Evaluate whether the model is adequate for the intended prediction task.

Reproducibility

  • Store the source code in a version-controlled repository.

  • Record material properties, initial conditions, and boundary conditions.

  • Document the numerical method and resolution.

  • Record software dependencies and execution instructions.

  • Preserve the dataset and analysis script.

Interpretation

Verification provides evidence that the numerical program solves the intended mathematical problem. Validation provides evidence about whether the mathematical model represents the real heat-transfer process. Reproducibility allows others to examine and repeat the investigation.

Uncertainty And Limitations

  • Verification cannot prove that the scientific model is realistic; it examines computational correctness.

  • Validation is limited by the quality, quantity, and range of available experimental evidence.

  • Reproducing the same result does not automatically establish that the result is scientifically correct.

  • Some stochastic simulations may produce different outputs unless random seeds and sampling procedures are controlled.

  • Hardware differences and numerical libraries may lead to small floating-point differences.

  • Some datasets cannot be publicly shared because of privacy, security, ownership, or licensing restrictions.

Common Mistakes

  • Assuming that a program is verified simply because it runs without crashing.

  • Confusing verification with validation.

  • Using the same data for both model calibration and independent validation without acknowledging the limitation.

  • Ignoring numerical convergence and stability.

  • Reporting results without documenting software versions and parameters.

  • Assuming that reproducibility guarantees scientific validity.

  • Ignoring uncertainty in experimental measurements and model parameters.

Practical Learning

  • Write unit tests for a program that calculates a known mathematical result.

  • Compare a numerical integration method with an analytical solution.

  • Repeat a simulation using different numerical step sizes and investigate convergence.

  • Compare a model prediction with a small independently collected dataset.

  • Place the program and its documentation in a version-controlled repository.

  • Recreate the computational result using a clean software environment and documented instructions.

Learning Objectives

  • Define verification, validation, and reproducibility.

  • Explain the different questions answered by these three quality principles.

  • Identify suitable verification and validation techniques.

  • Understand numerical accuracy, stability, convergence, and uncertainty.

  • Design a reproducible computational experiment.

  • Recognize why a computational result must be evaluated before scientific conclusions are drawn.

Assessment Questions

  • What is the difference between verification and validation?

  • Why does a successful software execution not prove that a scientific model is correct?

  • How can an analytical solution be used for code verification?

  • Why should independent data be used for model validation?

  • What information should be preserved to reproduce a scientific simulation?

  • Can a reproducible computational result still be scientifically wrong? Explain.

Example or application

A worked example has not yet been added.

Definition

Verification

Verification is the process of checking whether a computational implementation correctly solves the mathematical or computational model it is intended to represent.

Validation

Validation is the process of evaluating whether a mathematical and computational model adequately represents the real-world system for its intended use.

Reproducibility

Reproducibility is the ability to independently regenerate or examine computational results using sufficiently documented data, software, parameters, and procedures.

Combined Definition

Verification, Validation and Reproducibility are complementary principles used to establish computational correctness, scientific adequacy, and research transparency.

Purpose

  • To evaluate whether scientific software correctly implements its intended algorithms and mathematical models.

  • To determine whether a model adequately represents the real system under investigation.

  • To establish the reliability and transparency of computational results.

  • To identify numerical errors, model limitations, and uncertainty.

  • To make scientific investigations independently reviewable and repeatable.

Importance

  • Verification reduces the risk of drawing conclusions from incorrectly implemented algorithms or equations.

  • Validation provides evidence that computational models are suitable for their intended scientific applications.

  • Reproducibility enables independent checking, collaboration, and long-term scientific transparency.

  • These principles are especially important in safety-critical engineering, climate research, medicine, aerospace, and other fields where computational predictions can influence important decisions.

  • They help distinguish a functioning computational program from a scientifically trustworthy computational investigation.

Prerequisites

  • Introduction to Scientific Computing.

  • Mathematical Modelling in Scientific Computing.

  • Computational Workflows and Scientific Methodology.

  • Basic numerical analysis and software testing.

  • Introductory statistics and data analysis.

Related Topics

  • Computational Workflows and Scientific Methodology

  • Mathematical Modelling in Scientific Computing

  • Scientific Problem Formulation

  • Numerical Analysis

  • Scientific Computing Environments and Ecosystems

  • Uncertainty Quantification

Sources and verification

Sources are shown as provided in the project data. Not every source has yet been independently verified. Please report errors or missing citations.

  • ASME V&V standards for computational modelling and simulation
  • NASA-STD-7009: Standard for Models and Simulations
  • The Turing Way: A Handbook for Reproducible, Ethical and Collaborative Research
  • FAIR Guiding Principles for scientific data management and stewardship
  • Society for Industrial and Applied Mathematics (SIAM)

Suggest a correction or source · More in Scientific Computing