Designing probability samples to study treatment effect heterogeneity

Elizabeth Tipton, David S. Yeager, Ronaldo Iachan, Barbara Schneider

Research output: Chapter in Book/Report/Conference proceedingChapter

22 Scopus citations


This chapter explains a new approach that survey samplers can use when designing probability samples for survey experiments where there is a possibility of treatment heterogeneity. It begins by explaining why probability samples are preferred to nonprobability samples for estimating two quantities (or estimands): population average treatment effects and treatment effects within subgroups. The chapter furthermore explains why typical probability sampling methods that optimize statistical power for the average effect in a population do not necessarily optimize statistical power for the subgroup effects of interest - especially when one-s interest is in estimating effects within a rare subgroup. Next, it explains why even large, well-constructed, highly representative probability samples with randomized treatments can produce confounded analyses of differences across subgroups. The chapter illustrates the proposed approach using an empirical case study of a survey-administered behavioral science intervention: The US National Study of Learning Mindsets.

Original languageEnglish (US)
Title of host publicationExperimental Methods in Survey Research
Subtitle of host publicationTechniques that Combine Random Sampling with Random Assignment
Number of pages22
ISBN (Electronic)9781119083771
ISBN (Print)9781119083740
StatePublished - Sep 30 2019


  • Empirical case study
  • Population average treatment effects
  • Probability samples
  • Randomized treatment
  • Subgroup treatment effects
  • Survey experiments
  • Survey-administered behavioral science intervention
  • Treatment heterogeneity
  • US national study of learning mindsets

ASJC Scopus subject areas

  • General Social Sciences


Dive into the research topics of 'Designing probability samples to study treatment effect heterogeneity'. Together they form a unique fingerprint.

Cite this