Effect estimates of COVID-19 non-pharmaceutical interventions are non-robust and highly model-dependent

Vincent Chin; John P.A. Ioannidis; Martin A. Tanner; Sally Cripps

doi:10.1016/j.jclinepi.2021.03.014

Effect estimates of COVID-19 non-pharmaceutical interventions are non-robust and highly model-dependent

Vincent Chin, John P.A. Ioannidis^*, Martin A. Tanner, Sally Cripps

^*Corresponding author for this work

Statistics and Data Science

Research output: Contribution to journal › Article › peer-review

44 Scopus citations

Abstract

Objective: To compare the inference regarding the effectiveness of the various non-pharmaceutical interventions (NPIs) for COVID-19 obtained from different SIR models. Study design and setting: We explored two models developed by Imperial College that considered only NPIs without accounting for mobility (model 1) or only mobility (model 2), and a model accounting for the combination of mobility and NPIs (model 3). Imperial College applied models 1 and 2 to 11 European countries and to the USA, respectively. We applied these models to 14 European countries (original 11 plus another 3), over two different time horizons. Results: While model 1 found that lockdown was the most effective measure in the original 11 countries, model 2 showed that lockdown had little or no benefit as it was typically introduced at a point when the time-varying reproduction number was already very low. Model 3 found that the simple banning of public events was beneficial, while lockdown had no consistent impact. Based on Bayesian metrics, model 2 was better supported by the data than either model 1 or model 3 for both time horizons. Conclusion: Inferences on effects of NPIs are non-robust and highly sensitive to model specification. In the SIR modeling framework, the impacts of lockdown are uncertain and highly model-dependent.

Original language	English (US)
Pages (from-to)	96-132
Number of pages	37
Journal	Journal of Clinical Epidemiology
Volume	136
DOIs	https://doi.org/10.1016/j.jclinepi.2021.03.014
State	Published - Aug 2021

Funding

We congratulate the Imperial College Response Team for sharing openly the code for their models and for the overall transparency of their work that has allowed performing these analyses. We thank Hadi Ashfar for his suggestions to improve the computational efficiency of the HMC scheme. We also thank Jack Wood for his help in the construction of Table A.3. We especially thank the three reviewers and the Editor for their highly thoughtful and deep comments which greatly improved the quality of this paper. We acknowledge the Sydney Informatics Hub and the University of Sydney's high performance computing cluster Artemis for providing the high performance computing resources that have contributed to the research results reported within this paper. Funding: None.

Keywords

Bayesian statistics
COVID-19
Information criteria
Model comparison
Non-pharmaceutical interventions
SIR models

ASJC Scopus subject areas

Epidemiology

Access to Document

10.1016/j.jclinepi.2021.03.014

Cite this

@article{60a2c8af8f074d31961ba607a3b18a9a,

title = "Effect estimates of COVID-19 non-pharmaceutical interventions are non-robust and highly model-dependent",

abstract = "Objective: To compare the inference regarding the effectiveness of the various non-pharmaceutical interventions (NPIs) for COVID-19 obtained from different SIR models. Study design and setting: We explored two models developed by Imperial College that considered only NPIs without accounting for mobility (model 1) or only mobility (model 2), and a model accounting for the combination of mobility and NPIs (model 3). Imperial College applied models 1 and 2 to 11 European countries and to the USA, respectively. We applied these models to 14 European countries (original 11 plus another 3), over two different time horizons. Results: While model 1 found that lockdown was the most effective measure in the original 11 countries, model 2 showed that lockdown had little or no benefit as it was typically introduced at a point when the time-varying reproduction number was already very low. Model 3 found that the simple banning of public events was beneficial, while lockdown had no consistent impact. Based on Bayesian metrics, model 2 was better supported by the data than either model 1 or model 3 for both time horizons. Conclusion: Inferences on effects of NPIs are non-robust and highly sensitive to model specification. In the SIR modeling framework, the impacts of lockdown are uncertain and highly model-dependent.",

keywords = "Bayesian statistics, COVID-19, Information criteria, Model comparison, Non-pharmaceutical interventions, SIR models",

author = "Vincent Chin and Ioannidis, {John P.A.} and Tanner, {Martin A.} and Sally Cripps",

note = "Publisher Copyright: {\textcopyright} 2021 Elsevier Inc.",

year = "2021",

month = aug,

doi = "10.1016/j.jclinepi.2021.03.014",

language = "English (US)",

volume = "136",

pages = "96--132",

journal = "Journal of Clinical Epidemiology",

issn = "0895-4356",

publisher = "Elsevier Inc.",

}

TY - JOUR

T1 - Effect estimates of COVID-19 non-pharmaceutical interventions are non-robust and highly model-dependent

AU - Chin, Vincent

AU - Ioannidis, John P.A.

AU - Tanner, Martin A.

AU - Cripps, Sally

PY - 2021/8

Y1 - 2021/8

N2 - Objective: To compare the inference regarding the effectiveness of the various non-pharmaceutical interventions (NPIs) for COVID-19 obtained from different SIR models. Study design and setting: We explored two models developed by Imperial College that considered only NPIs without accounting for mobility (model 1) or only mobility (model 2), and a model accounting for the combination of mobility and NPIs (model 3). Imperial College applied models 1 and 2 to 11 European countries and to the USA, respectively. We applied these models to 14 European countries (original 11 plus another 3), over two different time horizons. Results: While model 1 found that lockdown was the most effective measure in the original 11 countries, model 2 showed that lockdown had little or no benefit as it was typically introduced at a point when the time-varying reproduction number was already very low. Model 3 found that the simple banning of public events was beneficial, while lockdown had no consistent impact. Based on Bayesian metrics, model 2 was better supported by the data than either model 1 or model 3 for both time horizons. Conclusion: Inferences on effects of NPIs are non-robust and highly sensitive to model specification. In the SIR modeling framework, the impacts of lockdown are uncertain and highly model-dependent.

AB - Objective: To compare the inference regarding the effectiveness of the various non-pharmaceutical interventions (NPIs) for COVID-19 obtained from different SIR models. Study design and setting: We explored two models developed by Imperial College that considered only NPIs without accounting for mobility (model 1) or only mobility (model 2), and a model accounting for the combination of mobility and NPIs (model 3). Imperial College applied models 1 and 2 to 11 European countries and to the USA, respectively. We applied these models to 14 European countries (original 11 plus another 3), over two different time horizons. Results: While model 1 found that lockdown was the most effective measure in the original 11 countries, model 2 showed that lockdown had little or no benefit as it was typically introduced at a point when the time-varying reproduction number was already very low. Model 3 found that the simple banning of public events was beneficial, while lockdown had no consistent impact. Based on Bayesian metrics, model 2 was better supported by the data than either model 1 or model 3 for both time horizons. Conclusion: Inferences on effects of NPIs are non-robust and highly sensitive to model specification. In the SIR modeling framework, the impacts of lockdown are uncertain and highly model-dependent.

KW - Bayesian statistics

KW - COVID-19

KW - Information criteria

KW - Model comparison

KW - Non-pharmaceutical interventions

KW - SIR models

UR - http://www.scopus.com/inward/record.url?scp=85103956888&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85103956888&partnerID=8YFLogxK

U2 - 10.1016/j.jclinepi.2021.03.014

DO - 10.1016/j.jclinepi.2021.03.014

M3 - Article

C2 - 33781862

AN - SCOPUS:85103956888

SN - 0895-4356

VL - 136

SP - 96

EP - 132

JO - Journal of Clinical Epidemiology

JF - Journal of Clinical Epidemiology

ER -

Effect estimates of COVID-19 non-pharmaceutical interventions are non-robust and highly model-dependent

Abstract

Funding

Keywords

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this