Exploring treatment impact heterogeneity across sites: Challenges and opportunities for early childhood researchers

Terri J. Sabol*, Dana McCoy, Kathryn Gonzalez, Luke Miratrix, Larry Hedges, Jessaca K. Spybrook, Christina Weiland

*Corresponding author for this work

Research output: Contribution to journalArticlepeer-review

6 Scopus citations

Abstract

The rise of multi-site, field-based trials in early childhood research coupled with advances in statistics offer an unprecedented opportunity to understand how context affects children's wellbeing. In the current study, we chart our journey in exploring heterogeneity in the treatment effects of an existing large-scale evaluation to provide guidance for early childhood researchers interested in studying treatment impact variation. To do so, we employ data from a professional development intervention implemented in early childhood education programs across 9 U.S. cities. We generate three broad lessons on the challenges and opportunities in examining treatment impacts across sites. First, we find that using the right statistical approach – namely fixed intercepts, random coefficient (FIRC) modeling – is critical for generating accurate estimates of cross-site variation. Second, we find that measures that traditionally have been associated with average treatment effects (e.g., program dosage) can be used to predict treatment impact. Third, we acknowledge trade-offs in statistical power for detecting average treatment effects versus treatment impact variation and discuss the implications of these trade-offs for the future design of early childhood evaluations. Findings are discussed in terms of how treatment impact variation has the potential to advance early childhood research.

Original languageEnglish (US)
Pages (from-to)14-26
Number of pages13
JournalEarly Childhood Research Quarterly
Volume58
DOIs
StatePublished - Jan 1 2022

Funding

This work was supported by the Institute of Education Sciences , U.S. Department of Education through Grant No. R305A160013 . The original professional study was supported by the Institute of Education Sciences, U.S. Department of Education through Grant R305A060021 to the University of Virginia. The opinions expressed are those of the authors and do not represent views of the U.S. Department of Education. We are grateful for the significant contributions from Drs. Robert Pianta, Jason Downer, Candice Odgers, and Phil Gagne. In addition, we would like to acknowledge Emily Hanno, Andrea Busby, Olivia Healy, Wendy Wei, Sarah Guminski, and Allie Cooperman for their feedback and support.

Keywords

  • Child development
  • Early childhood education
  • Treatment impact variation

ASJC Scopus subject areas

  • Education
  • Developmental and Educational Psychology
  • Sociology and Political Science

Fingerprint

Dive into the research topics of 'Exploring treatment impact heterogeneity across sites: Challenges and opportunities for early childhood researchers'. Together they form a unique fingerprint.

Cite this