Towards a better understanding of model validation metrics

Yu Liu, Wei Chen*, Paul Arendt, Hong Zhong Huang

*Corresponding author for this work

Research output: Chapter in Book/Report/Conference proceedingConference contribution

10 Scopus citations

Abstract

Model validation metrics have been developed to provide a quantitative measure that characterizes the agreement between predictions and observations. The metrics become useful for model selection when alternative models are being considered. Additionally, they are utilized when performing accuracy assessments before and after model improvement. Due to the various sources of uncertainties in both computer simulations and physical experiments, model validation must be conducted based on stochastic characteristics. Currently there is no unified validation metric that is widely accepted. In this paper, we present a classification of validation metrics based on their key characteristics along with a discussion of the desired features. In the category of stochastic validation with the consideration of uncertainty in both predictions and physical experiments, four main types of metrics, namely classical hypothesis testing, Bayes factor, frequentist's metric, and area metric, are examined to provide a better understanding of the pros and cons of each. Numerical examples are used to illustrate the differences in these metrics, and to examine how sensitive these metrics are with respect to the experimental data size, uncertainty from measurement error, and uncertainty in unknown model parameters. The insight gained from this work provides useful guidelines for choosing the appropriate metric in model validation.

Original languageEnglish (US)
Title of host publication13th AIAA/ISSMO Multidisciplinary Analysis and Optimization Conference 2010
DOIs
StatePublished - 2010
Event13th AIAA/ISSMO Multidisciplinary Analysis and Optimization Conference, MAO 2010 - Ft. Worth, TX, United States
Duration: Sep 13 2010Sep 15 2010

Publication series

Name13th AIAA/ISSMO Multidisciplinary Analysis and Optimization Conference 2010

Other

Other13th AIAA/ISSMO Multidisciplinary Analysis and Optimization Conference, MAO 2010
Country/TerritoryUnited States
CityFt. Worth, TX
Period9/13/109/15/10

Funding

The grant supports from the US National Science Foundation (CMMI-0928320), the China Scholarship Council and the Fundamental Research Funds for the Central Universities are greatly acknowledged. The views expressed are those of the authors and do not necessarily reflect the views of the sponsors.

ASJC Scopus subject areas

  • Aerospace Engineering
  • Mechanical Engineering

Fingerprint

Dive into the research topics of 'Towards a better understanding of model validation metrics'. Together they form a unique fingerprint.

Cite this