How Generalizable Is Your Experiment? An Index for Comparing Experimental Samples and Populations

Elizabeth Tipton*

*Corresponding author for this work

Research output: Contribution to journalArticlepeer-review

45 Scopus citations

Abstract

Although a large-scale experiment can provide an estimate of the average causal impact for a program, the sample of sites included in the experiment is often not drawn randomly from the inference population of interest. In this article, we provide a generalizability index that can be used to assess the degree of similarity between the sample of units in an experiment and one or more inference populations on a set of selected covariates. The index takes values between 0 and 1 and indicates both when a sample is like a miniature of the population and how well reweighting methods may perform when differences exist. Results of simulation studies are provided that develop rules of thumb for interpretation as well as an example.

Original languageEnglish (US)
Pages (from-to)478-501
Number of pages24
JournalJournal of Educational and Behavioral Statistics
Volume39
Issue number6
DOIs
StatePublished - Dec 20 2014

Keywords

  • causal inference
  • experiment
  • external validity
  • generalizability
  • index

ASJC Scopus subject areas

  • Education
  • Social Sciences (miscellaneous)

Fingerprint Dive into the research topics of 'How Generalizable Is Your Experiment? An Index for Comparing Experimental Samples and Populations'. Together they form a unique fingerprint.

Cite this