Abstract
Model validation metrics have been developed to provide a quantitative measure that characterizes the agreement between predictions and observations. The metrics become useful for model selection when alternative models are being considered. Additionally, they are utilized when performing accuracy assessments before and after model improvement. Due to the various sources of uncertainties in both computer simulations and physical experiments, model validation must be conducted based on stochastic characteristics. Currently there is no unified validation metric that is widely accepted. In this paper, we present a classification of validation metrics based on their key characteristics along with a discussion of the desired features. In the category of stochastic validation with the consideration of uncertainty in both predictions and physical experiments, four main types of metrics, namely classical hypothesis testing, Bayes factor, frequentist's metric, and area metric, are examined to provide a better understanding of the pros and cons of each. Numerical examples are used to illustrate the differences in these metrics, and to examine how sensitive these metrics are with respect to the experimental data size, uncertainty from measurement error, and uncertainty in unknown model parameters. The insight gained from this work provides useful guidelines for choosing the appropriate metric in model validation.
Original language | English (US) |
---|---|
Title of host publication | 13th AIAA/ISSMO Multidisciplinary Analysis and Optimization Conference 2010 |
DOIs | |
State | Published - 2010 |
Event | 13th AIAA/ISSMO Multidisciplinary Analysis and Optimization Conference, MAO 2010 - Ft. Worth, TX, United States Duration: Sep 13 2010 → Sep 15 2010 |
Publication series
Name | 13th AIAA/ISSMO Multidisciplinary Analysis and Optimization Conference 2010 |
---|
Other
Other | 13th AIAA/ISSMO Multidisciplinary Analysis and Optimization Conference, MAO 2010 |
---|---|
Country/Territory | United States |
City | Ft. Worth, TX |
Period | 9/13/10 → 9/15/10 |
Funding
The grant supports from the US National Science Foundation (CMMI-0928320), the China Scholarship Council and the Fundamental Research Funds for the Central Universities are greatly acknowledged. The views expressed are those of the authors and do not necessarily reflect the views of the sponsors.
ASJC Scopus subject areas
- Aerospace Engineering
- Mechanical Engineering