Three Conditions under Which Experiments and Observational Studies Produce Comparable Causal Estimates: New Findings from Within-Study Comparisons

Thomas D. Cook*, William R. Shadish, Vivian C. Wong

*Corresponding author for this work

Research output: Chapter in Book/Report/Conference proceedingChapter

Abstract

This paper analyzes 12 recent within-study comparisons contrasting causal estimates from a randomized experiment with those from an observational study sharing the same treatment group. The aim is to test whether different causal estimates result when a counterfactual group is formed, either with or without random assignment, and when statistical adjustments for selection are made in the group from which random assignment is absent. We identify three studies comparing experiments and regression-discontinuity (RD) studies. They produce quite comparable causal estimates at points around the RD cutoff. We identify three other studies where the quasi-experiment involves careful intact group matching on the pretest. Despite the logical possibility of hidden bias in this instance, all three cases also reproduce their experimental estimates, especially if the match is geographically local. We then identify two studies where the treatment and nonrandomized comparison groups manifestly differ at pretest but where the selection process into treatment is completely or very plausibly known. Here too, experimental results are recreated. Two of the remaining studies result in correspondent experimental and nonexperimental results under some circumstances but not others, while two others produce different experimental and nonexperimental estimates, though in each case the observational study was poorly designed and analyzed. Such evidence is more promising than what was achieved in past within-study comparisons, most involving job training. Reasons for this difference are discussed.

Original languageEnglish (US)
Title of host publicationSocial Experimentation, Program Evaluation, and Public Policy
PublisherBlackwell Publishing Ltd
Pages92-118
Number of pages27
ISBN (Print)9781405193931
DOIs
StatePublished - Apr 21 2009

Keywords

  • Case I: Contrasting Experimental and Regression-Discontinuity Design Results
  • Case II: Contrasting Experiments and Observational Studies with Matched Intact Comparison Groups
  • Case III: Contrasting Experiments and Observational Studies with Manifestly Different Populations but Where the Selection Process into Treatment is Known
  • Criteria for Evaluating Within-Study Comparisons
  • More Ambiguous Within-Study Comparison Cases

ASJC Scopus subject areas

  • Social Sciences(all)

Fingerprint Dive into the research topics of 'Three Conditions under Which Experiments and Observational Studies Produce Comparable Causal Estimates: New Findings from Within-Study Comparisons'. Together they form a unique fingerprint.

  • Cite this

    Cook, T. D., Shadish, W. R., & Wong, V. C. (2009). Three Conditions under Which Experiments and Observational Studies Produce Comparable Causal Estimates: New Findings from Within-Study Comparisons. In Social Experimentation, Program Evaluation, and Public Policy (pp. 92-118). Blackwell Publishing Ltd. https://doi.org/10.1002/9781444307399.ch8