New validation metrics for models with multiple correlated responses

Wei Li, Wei Chen*, Zhen Jiang, Zhenzhou Lu, Yu Liu

*Corresponding author for this work

Research output: Contribution to journalArticlepeer-review

53 Scopus citations


Validating models with correlated multivariate outputs involves the comparison of multiple stochastic quantities. Considering both uncertainty and correlations among multiple responses from model and physical observations imposes challenges. Existing marginal comparison methods and the hypothesis testing-based methods either ignore correlations among responses or only reach Boolean conclusions (yes or no) without accounting for the amount of discrepancy between a model and the underlying reality. A new validation metric is needed to quantitatively characterize the overall agreement of multiple responses considering correlations among responses and uncertainty in both model predictions and physical observations. In this paper, by extending the concept of "area metric" and the "u-pooling method" developed for validating a single response, we propose new model validation metrics for validating correlated multiple responses using the multivariate probability integral transformation (PIT). One new metric is the PIT area metric for validating multi-responses at a single validation site. The other is the t-pooling metric that allows for pooling observations of multiple responses observed at multiple validation sites to assess the global predictive capability. The proposed metrics have many favorable properties that are well suited for validation assessment of models with correlated responses. The two metrics are examined and compared with the direct area metric and the marginal u-pooling method respectively through numerical case studies and an engineering example to illustrate their validity and potential benefits.

Original languageEnglish (US)
Pages (from-to)1-11
Number of pages11
JournalReliability Engineering and System Safety
StatePublished - Jul 2014


  • Area metric
  • Correlation
  • Model validation
  • Multi-response
  • Multivariate probability integral transformation
  • Uncertainty

ASJC Scopus subject areas

  • Safety, Risk, Reliability and Quality
  • Industrial and Manufacturing Engineering


Dive into the research topics of 'New validation metrics for models with multiple correlated responses'. Together they form a unique fingerprint.

Cite this