Learning as we go – An examination of the statistical accuracy of COVID-19 daily death count predictions

OBJECTIVE: This paper provides a formal evaluation of the predictive performance of a model (and updates) developed by the Institute for Health Metrics and Evaluation (IHME) for predicting daily deaths attributed to COVID-19 for the United States. STUDY DESIGN: To assess the accuracy of the IHME models, we examine both forecast accuracy, as well as the predictive performance of the 95% prediction intervals (PI). RESULTS: The initial model underestimates the uncertainty surrounding the number of daily deaths. Specifically, the true number of next day deaths fell outside the IHME prediction intervals as much as 76% of the time, in comparison to the expected value of 5%. Regarding the updated models, our analyses indicate that the April models show little, if any, improvement in the accuracy of the point estimate predictions. Moreover, while we observe a larger percentage of states having actual values lying inside the 95% PI’s, this observation may be attributed to the widening of the PI’s. A major revised model in early May did result in a decrease in the estimated model uncertainty, albeit at the expense of poorer coverage probability. CONCLUSION: Our analysis calls into question the usefulness of the predictions to drive policy making and resource allocation.