Vote-counting methods in research synthesis

Reviews of research usually rely on counts of the number of times the treatment group mean exceeds the control group mean by an amount that is statistically significant. The treatment is said to have a positive effect if the proportion of such positive significant results is large. This procedure is shown to have extremely low power for the combination of treatment-effect sizes and sample sizes usually found in social science research. Surprisingly, the power of this procedure decreases as the number of studies reviewed increases. Three alternative counting procedures that permit estimation of the standardized mean difference between treatment and control groups (effect size) are described. Methods for obtaining confidence intervals for the effect size are also presented. For 1 procedure, the effect size can be estimated even when the sample consists only of studies that have statistically significant results. (7 ref) (PsycINFO Database Record (c) 2006 APA, all rights reserved).