Learning as we go – An examination of the statistical accuracy of COVID-19 daily death count predictions

Roman Marchant, Noelle I. Samia, Ori Rosen, Martin A. Tanner, Sally Cripps*

*Corresponding author for this work

Research output: Contribution to journalArticlepeer-review

Abstract

OBJECTIVE: This paper provides a formal evaluation of the predictive performance of a model (and updates) developed by the Institute for Health Metrics and Evaluation (IHME) for predicting daily deaths attributed to COVID-19 for the United States. STUDY DESIGN: To assess the accuracy of the IHME models, we examine both forecast accuracy, as well as the predictive performance of the 95% prediction intervals (PI). RESULTS: The initial model underestimates the uncertainty surrounding the number of daily deaths. Specifically, the true number of next day deaths fell outside the IHME prediction intervals as much as 76% of the time, in comparison to the expected value of 5%. Regarding the updated models, our analyses indicate that the April models show little, if any, improvement in the accuracy of the point estimate predictions. Moreover, while we observe a larger percentage of states having actual values lying inside the 95% PI’s, this observation may be attributed to the widening of the PI’s. A major revised model in early May did result in a decrease in the estimated model uncertainty, albeit at the expense of poorer coverage probability. CONCLUSION: Our analysis calls into question the usefulness of the predictions to drive policy making and resource allocation.

Original languageEnglish (US)
JournalUnknown Journal
StatePublished - Apr 8 2020

Keywords

  • COVID-19
  • Decision making under uncertainty
  • Forecast accuracy
  • Model calibration
  • Public health resource allocation
  • Uncertainty quantification

ASJC Scopus subject areas

  • General

Fingerprint Dive into the research topics of 'Learning as we go – An examination of the statistical accuracy of COVID-19 daily death count predictions'. Together they form a unique fingerprint.

Cite this