Estimating Effect Size under Publication Bias: Small Sample Properties and Robustness of a Random Effects Selection Model

Larry V. Hedges*, Jack L. Vevea

*Corresponding author for this work

Research output: Contribution to journalArticlepeer-review

122 Scopus citations

Abstract

When there is publication bias, studies yielding large p values, and hence small effect estimates, are less likely to be published, which leads to biased estimates of effects in meta-analysis. We investigate a selection model based on one-tailed p values in the context of a random effects model. The procedure both models the selection process and corrects for the consequences of selection on estimates of the mean and variance of effect parameters. A test of the statistical significance of selection is also provided. The small sample properties of the method are evaluated by means of simulations, and the asymptotic theory is found to be reasonably accurate under correct model specification and plausible conditions. The method substantially reduces bias due to selection when model specification is correct, but the variance of estimates is increased; thus mean squared error is reduced only when selection produces substantial bias. The robustness of the method to violations of assumptions about the form of the distribution of the random effects is also investigated via simulation, and the model-corrected estimates of the mean effect are generally found to be much less biased than the uncorrected estimates. The significance test for selection bias, however, is found to be highly nonrobust, rejecting at up to 10 times the nominal rate when there is no selection but the distribution of the effects is incorrectly specified.

Original languageEnglish (US)
Pages (from-to)299-332
Number of pages34
JournalJournal of Educational and Behavioral Statistics
Volume21
Issue number4
DOIs
StatePublished - 1996

Keywords

  • File drawer problem
  • Meta-analysis
  • Publication bias
  • Random effects models
  • Robustness
  • Selection models

ASJC Scopus subject areas

  • Education
  • Social Sciences (miscellaneous)

Fingerprint Dive into the research topics of 'Estimating Effect Size under Publication Bias: Small Sample Properties and Robustness of a Random Effects Selection Model'. Together they form a unique fingerprint.

Cite this