TY - JOUR
T1 - THE INTERNAL AND EXTERNAL VALIDITY OF THE REGRESSION DISCONTINUITY DESIGN
T2 - A META-ANALYSIS OF 15 WITHIN-STUDY COMPARISONS
AU - Chaplin, Duncan D.
AU - Cook, Thomas D.
AU - Zurovac, Jelena
AU - Coopersmith, Jared S.
AU - Finucane, Mariel M.
AU - Vollmer, Lauren N.
AU - Morris, Rebecca E.
N1 - Funding Information:
This research was supported by National Science Foundation Grants: DRL-1228866 and DGE-1544301. The authors would also like to thank Austin Nichols, Phil Gleason, Steve Glazerman, Heinrich Hock, Hanley Chiang, Larry Hedges, Peter Steiner, and reviewers from this journal for valuable advice given during the writing of this paper.
PY - 2018/3/1
Y1 - 2018/3/1
N2 - Theory predicts that regression discontinuity (RD) provides valid causal inference at the cutoff score that determines treatment assignment. One purpose of this paper is to test RD's internal validity across 15 studies. Each of them assesses the correspondence between causal estimates from an RD study and a randomized control trial (RCT) when the estimates are made at the same cutoff point where they should not differ asymptotically. However, statistical error, imperfect design implementation, and a plethora of different possible analysis options, mean that they might nonetheless differ. We test whether they do, assuming that the bias potential is greater with RDs than RCTs. A second purpose of this paper is to investigate the external validity of RD by exploring how the size of the bias estimates varies across the 15 studies, for they differ in their settings, interventions, analyses, and implementation details. Both Bayesian and frequentist meta-analysis methods show that the RD bias is below 0.01 standard deviations on average, indicating RD's high internal validity. When the study-specific estimates are shrunken to capitalize on the information the other studies provide, all the RD causal estimates fall within 0.07 standard deviations of their RCT counterparts, now indicating high external validity. With unshrunken estimates, the mean RD bias is still essentially zero, but the distribution of RD bias estimates is less tight, especially with smaller samples and when parametric RD analyses are used.
AB - Theory predicts that regression discontinuity (RD) provides valid causal inference at the cutoff score that determines treatment assignment. One purpose of this paper is to test RD's internal validity across 15 studies. Each of them assesses the correspondence between causal estimates from an RD study and a randomized control trial (RCT) when the estimates are made at the same cutoff point where they should not differ asymptotically. However, statistical error, imperfect design implementation, and a plethora of different possible analysis options, mean that they might nonetheless differ. We test whether they do, assuming that the bias potential is greater with RDs than RCTs. A second purpose of this paper is to investigate the external validity of RD by exploring how the size of the bias estimates varies across the 15 studies, for they differ in their settings, interventions, analyses, and implementation details. Both Bayesian and frequentist meta-analysis methods show that the RD bias is below 0.01 standard deviations on average, indicating RD's high internal validity. When the study-specific estimates are shrunken to capitalize on the information the other studies provide, all the RD causal estimates fall within 0.07 standard deviations of their RCT counterparts, now indicating high external validity. With unshrunken estimates, the mean RD bias is still essentially zero, but the distribution of RD bias estimates is less tight, especially with smaller samples and when parametric RD analyses are used.
UR - http://www.scopus.com/inward/record.url?scp=85041702051&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85041702051&partnerID=8YFLogxK
U2 - 10.1002/pam.22051
DO - 10.1002/pam.22051
M3 - Article
AN - SCOPUS:85041702051
VL - 37
SP - 403
EP - 429
JO - Journal of Policy Analysis and Management
JF - Journal of Policy Analysis and Management
SN - 0276-8739
IS - 2
ER -