A Design-Based Approach to Improve External Validity in Welfare Policy Evaluations

Elizabeth Tipton*, Laura R. Peck

*Corresponding author for this work

Research output: Contribution to journalArticlepeer-review

9 Scopus citations

Abstract

Background: Large-scale randomized experiments are important for determining how policy interventions change average outcomes. Researchers have begun developing methods to improve the external validity of these experiments. One new approach is a balanced sampling method for site selection, which does not require random sampling and takes into account the practicalities of site recruitment including high nonresponse. Method: The goal of balanced sampling is to develop a strategic sample selection plan that results in a sample that is compositionally similar to a well-defined inference population. To do so, a population frame is created and then divided into strata, which “focuses” recruiters on specific subpopulations. Units within these strata are then ranked, thus identifying “replacements” similar to sites that can be recruited when the ideal site refuses to participate in the experiment. Result: In this article, we consider how a balanced sample strategic site selection method might be implemented in a welfare policy evaluation. Conclusion: We find that simply developing a population frame can be challenging, with three possible and reasonable options arising in the welfare policy arena. Using relevant study-specific contextual variables, we craft a recruitment plan that considers nonresponse.

Original languageEnglish (US)
Pages (from-to)326-356
Number of pages31
JournalEvaluation Review
Volume41
Issue number4
DOIs
StatePublished - Aug 1 2017

Keywords

  • content area
  • job training
  • methodological development

ASJC Scopus subject areas

  • Arts and Humanities (miscellaneous)
  • Social Sciences(all)

Fingerprint Dive into the research topics of 'A Design-Based Approach to Improve External Validity in Welfare Policy Evaluations'. Together they form a unique fingerprint.

Cite this