TY - JOUR
T1 - Comparative Regression Discontinuity
T2 - A Stress Test With Small Samples
AU - Kisbu-Sakarya, Yasemin
AU - Cook, Thomas D.
AU - Tang, Yang
AU - Clark, M. H.
N1 - Funding Information:
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: Research reported in this article was supported by the National Science Foundation PRIME Grant DRL-1228866.
Publisher Copyright:
© The Author(s) 2018.
PY - 2018/2/1
Y1 - 2018/2/1
N2 - Compared to the randomized experiment (RE), the regression discontinuity design (RDD) has three main limitations: (1) In expectation, its results are unbiased only at the treatment cutoff and not for the entire study population; (2) it is less efficient than the RE and so requires more cases for the same statistical power; and (3) it requires correctly specifying the functional form that relates the assignment and outcome variables. One way to overcome these limitations might be to add a no-treatment functional form to the basic RDD and including it in the outcome analysis as a comparison function rather than as a covariate to increase power. Doing this creates a comparative regression discontinuity design (CRD). It has three untreated regression lines. Two are in the untreated segment of the RDD—the usual RDD one and the added untreated comparison function—while the third is in the treated RDD segment. Also observed is the treated regression line in the treated segment. Recent studies comparing RE, RDD, and CRD causal estimates have found that CRD reduces imprecision compared to RDD and also produces valid causal estimates at the treatment cutoff and also along all the rest of the assignment variable. The present study seeks to replicate these results, but with considerably smaller sample sizes. The power difference between RDD and CRD is replicated, but not the bias results either at the treatment cutoff or away from it. We conclude that CRD without large samples can be dangerous.
AB - Compared to the randomized experiment (RE), the regression discontinuity design (RDD) has three main limitations: (1) In expectation, its results are unbiased only at the treatment cutoff and not for the entire study population; (2) it is less efficient than the RE and so requires more cases for the same statistical power; and (3) it requires correctly specifying the functional form that relates the assignment and outcome variables. One way to overcome these limitations might be to add a no-treatment functional form to the basic RDD and including it in the outcome analysis as a comparison function rather than as a covariate to increase power. Doing this creates a comparative regression discontinuity design (CRD). It has three untreated regression lines. Two are in the untreated segment of the RDD—the usual RDD one and the added untreated comparison function—while the third is in the treated RDD segment. Also observed is the treated regression line in the treated segment. Recent studies comparing RE, RDD, and CRD causal estimates have found that CRD reduces imprecision compared to RDD and also produces valid causal estimates at the treatment cutoff and also along all the rest of the assignment variable. The present study seeks to replicate these results, but with considerably smaller sample sizes. The power difference between RDD and CRD is replicated, but not the bias results either at the treatment cutoff or away from it. We conclude that CRD without large samples can be dangerous.
KW - causal inference
KW - comparative regression discontinuity
KW - regression discontinuity design
KW - within-study comparison
UR - http://www.scopus.com/inward/record.url?scp=85047930373&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85047930373&partnerID=8YFLogxK
U2 - 10.1177/0193841X18776881
DO - 10.1177/0193841X18776881
M3 - Article
C2 - 29852743
AN - SCOPUS:85047930373
VL - 42
SP - 111
EP - 143
JO - Evaluation Review
JF - Evaluation Review
SN - 0193-841X
IS - 1
ER -