Designing probability samples to study treatment effect heterogeneity

This chapter explains a new approach that survey samplers can use when designing probability samples for survey experiments where there is a possibility of treatment heterogeneity. It begins by explaining why probability samples are preferred to nonprobability samples for estimating two quantities (or estimands): population average treatment effects and treatment effects within subgroups. The chapter furthermore explains why typical probability sampling methods that optimize statistical power for the average effect in a population do not necessarily optimize statistical power for the subgroup effects of interest - especially when one-s interest is in estimating effects within a rare subgroup. Next, it explains why even large, well-constructed, highly representative probability samples with randomized treatments can produce confounded analyses of differences across subgroups. The chapter illustrates the proposed approach using an empirical case study of a survey-administered behavioral science intervention: The US National Study of Learning Mindsets.