How much bias results if a quasi-experimental design combines local comparison groups, a pretest outcome measure and other covariates? A within study comparison of preschool effects.

Thomas D. Cook*, Naixin Zhu, Alice Klein, Prentice Starkey, Jaime Thomas

*Corresponding author for this work

Research output: Contribution to journalArticlepeer-review

11 Scopus citations


This study uses a within study comparison design (WSC) to conduct a novel test of how much causal bias results when researchers use a nonequivalent comparison group design type (NECGD) that combines: (a) a comparison group local to the treatment group; (b) a pretest measure of the study outcome; and (c) a rich set of 19 other multidimensional covariates. Most prior WSCs have dealt with the bias consequences of only 1 of these, revealing that each routinely reduces bias but does not necessarily eliminate it. Thus, a need exists to identify NECGDs that more robustly eliminate bias. This study is the first to examine how combining the 3 bias-control mechanisms above affects bias. The intervention we examine is a prekindergarten mathematics curriculum, for which a randomized control trial (RCT) produces a positive 1-year math effect. Final bias in the NECGD is assessed as the difference between its impact and that of the RCT when each design has the same intervention, outcome, and estimand. Over the many specifications we explore, NECGD bias is less than .10 standard deviations, indicating that minimal bias results when an NECGD combines all 3 design elements. The factorial design we use in this study also tests the bias associated with seven other NECGD types. Comparing the total pattern of results shows that the minimal bias when all 3 elements are combined is uniquely attributable to the locally chosen comparison group and not the availability of a pretest or other covariates. In actual research practice, it is impossible to predict in advance which design elements will affect bias by how much in any given application. So further research is needed to probe whether the simultaneous use of all three design elements achieves minimal bias dependably across diverse applications and not just in the preschool math context examined here. (PsycInfo Database Record (c) 2020 APA, all rights reserved)Translational Abstract—This study examines when nonexperiments might substitute for experiments that are done in real-world settings in order to learn what works to affect some socially valued outcome. The study probes whether a similar result is achieved in an experiment and in a nonexperiment that lacks the randomly formed control group of the experiment but that has instead a nonequivalent control group. However, this nonequivalent group is locally chosen to the treatment group in hopes of reducing the size of initial group differences and to match on whichever policies and practices are set locally. Moreover, the nonexperiment in question has a pretest measure of the study outcome and a “rich” set of other multidimensional measures with which to model whatever differences exist after the nonequivalent treatment and control groups have been selected from the same local pool. The study shows that the experimental and nonexperimental estimates are very close if the nonexperiment is defined in terms of local comparison group choice, a pretest measure of the study outcome and a rich set of other preintervention measures. This result now needs replicating. (PsycInfo Database Record (c) 2020 APA, all rights reserved)

Original languageEnglish (US)
Pages (from-to)726-746
Number of pages21
JournalPsychological methods
Issue number6
StatePublished - 2020


  • bias after local comparison group choice
  • causation
  • observational studies
  • use of pretest and of a rich set of other covariates
  • within-study comparison

ASJC Scopus subject areas

  • Psychology (miscellaneous)


Dive into the research topics of 'How much bias results if a quasi-experimental design combines local comparison groups, a pretest outcome measure and other covariates? A within study comparison of preschool effects.'. Together they form a unique fingerprint.

Cite this