Analyzing effect sizes: Fixed-effects models

A central question in research synthesis is whether methodological, contextual, or substantive differences in research studies are related to variation in effect-size parameters. Both fixed- and random-effects statistical methods are available for studying the variation in effects. The choice of which to use is sometimes a contentious issue in both meta-analysis as well as primary analysis of data. The choice of statistical procedures should primarily be determined by the kinds of inference the synthesist wishes to make. Two different inference models are available, sometimes called conditional and unconditional inference (see Hedges and Vevea 1998). The conditional model attempts to make inference about the relation between covariates and the effect-size parameters in the studies that are observed. In contrast, the unconditional model attempts to make inferences about the relation between covariates and the effect-size parameters in the population of studies from which the observed studies are a representative sample. Fixed-effects statistical procedures are well suited to drawing conditional inferences about the observed studies (see, for example, Hedges and Vevea 1998). Random- or mixed-effects statistical procedures are well suited to drawing unconditional inferences (inferences about the population of studies from which the observed studies are a sample. Fixed-effects statistical procedures may also be a reasonable choice when the number of studies is too small to support the effective use of mixed- or random-effects models. In this chapter, we present two general classes of fixed-effects models. One class is appropriate when the independent (study characteristic) variables are categorical. This class is analogous to the analysis of variance, but is adapted to the special characteristics of effect-size estimates. The second class is appropriate for either discrete or continuous independent (study characteristic) variables and therefore technically includes the first class as a special case. This second class is analogous to multiple regression analysis for effect sizes. In both cases, we describe the models along with procedures for estimation and hypothesis testing. Although some formulas for hand computation are given, we stress computation using widely available packaged computer programs. Tests of goodness of fit are given for each fixed-effect model. They test the notion that there is no more variability in the observed effect sizes than would be expected if all (100 percent) of the variation in effect size parameters is explained by the data analysis model. These tests can be conceived as tests of model specification. If a fixed-effects model explains all of the variation in effect-size parameters, the (fixed-effect) model is unquestionably appropriate. Models that are well specified can provide a strong basis for inference about effect sizes in fixed-effects models, but are not essential for inference from them. If differences between studies that lead to differences in effects are not regarded as random (for example, if they are regarded as consequences of purposeful design decisions) then fixed effects methods may be appropriate for the analysis. Similarly fixed-effects analyses are appropriate if the inferences desired are regarded as conditional - applying only to studies like those under examination.