Testing the Null Hypothesis in Meta-Analysis: A Comparison of Combined Probability and Confidence Interval Procedures

Combined significance tests (combined p values) and tests of the weighted mean effect size are both used to combine information across studies in meta-analysis. This article compares a combined significance test (the Stouffer test) with a test based on the weighted mean effect size as tests of the same null hypothesis. The tests are compared analytically in the case in which the within-group variances are known and compared through large-sample theory in the more usual case in which the variances are unknown. Generalizations suggested are then explored through a simulation study. This work demonstrates that the test based on the average effect size is usually more powerful than the Stouffer test unless there is a substantial negative correlation between within-study sample size and effect size. Thus the test based on the average effect size is generally preferable, and there is little reason to also calculate the Stouffer test.