Implications of Small Samples for Generalization: Adjustments and Rules of Thumb

Elizabeth Tipton*, Kelly Hallberg, Larry V. Hedges, Wendy Chan

*Corresponding author for this work

Research output: Contribution to journalArticlepeer-review

77 Scopus citations

Abstract

Background: Policy makers and researchers are frequently interested in understanding how effective a particular intervention may be for a specific population. One approach is to assess the degree of similarity between the sample in an experiment and the population. Another approach is to combine information from the experiment and the population to estimate the population average treatment effect (PATE). Method: Several methods for assessing the similarity between a sample and population currently exist as well as methods estimating the PATE. In this article, we investigate properties of six of these methods and statistics in the small sample sizes common in education research (i.e., 10–70 sites), evaluating the utility of rules of thumb developed from observational studies in the generalization case. Result: In small random samples, large differences between the sample and population can arise simply by chance and many of the statistics commonly used in generalization are a function of both sample size and the number of covariates being compared. The rules of thumb developed in observational studies (which are commonly applied in generalization) are much too conservative given the small sample sizes found in generalization. Conclusion: This article implies that sharp inferences to large populations from small experiments are difficult even with probability sampling. Features of random samples should be kept in mind when evaluating the extent to which results from experiments conducted on nonrandom samples might generalize.

Original languageEnglish (US)
Pages (from-to)472-505
Number of pages34
JournalEvaluation Review
Volume41
Issue number5
DOIs
StatePublished - Oct 1 2017

Keywords

  • content area
  • education
  • methodological development

ASJC Scopus subject areas

  • Arts and Humanities (miscellaneous)
  • General Social Sciences

Fingerprint

Dive into the research topics of 'Implications of Small Samples for Generalization: Adjustments and Rules of Thumb'. Together they form a unique fingerprint.

Cite this