Effect sizes in nested designs

Studies with nested designs are frequently used to evaluate the effects of social treatments, such as interventions, products, or technologies in education or public health. One common nested design assigns entire sites - often classrooms, schools, clinics, or communities - to the same treatment group, with different sites assigned to different treatments. Experiments with designs of this type are also called group randomized or cluster randomized designs because sites such as schools or communities correspond to statistical clusters. In experimental design terminology, these designs are designs involving clusters as nested factors. Nested factors are groupings of individuals that occur only within one treatment condition, such as schools or communities in designs that assign whole schools or communities to treatments. Assigning entire groups to receive the same treatments is sometimes done for the convenience of the investigator because it is easier to provide the same treatment to all individuals in a group. In other situations, group assignment is used to prevent contamination of one treatment group by elements of treatments assigned to others. In this case, physical separation of groups (for example, different schools or communities) minimizes contamination. In other situations, the treatment is an administrative or group-based manipulation, making it conceptually difficult to imagine how the treatment could be assigned at an individual level. For example, treatments that involve changing teacher behavior cannot easily be assigned to individual students, nor can administrative or procedural reforms in a school or clinic easily be assigned to individuals within a school or clinic. Although the analysis of cluster randomized designs has received considerable attention in the statistical literature (for example, Donner and Klar 2000; Raudenbush and Bryk 2002), the problem of representation of the results of cluster randomized trials (and the corresponding quasi-experiments) in the form of effect sizes and combining them across studies in meta-analyses have received less attention. Brenda Rooney and David Murray called attention to the problem of effect-size estimation in cluster randomized trials and suggested that neither the conventional estimates of effect sizes nor the formulas for their standard errors were appropriate in this situation (1996). In particular, standard errors of effect-size estimates will be profoundly underestimated if conventional formulas are used. Allan Donner and Neil Klar suggested that corrections for the effects of clustering should be introduced in meta-analyses of cluster randomized experiments (2002). Larry Hedges studied the problem of effect sizes in cluster randomized designs and computed their sampling distributions, including their standard errors (2007, in press). The standardized mean difference, which is widely used as an effect-size index in educational and psychological research, is defined as the ratio of a difference between treatment and control group means to a standard deviation. In designs where there is no nesting, that is, where there is no statistical clustering, the notion of standardized mean difference is often unambiguous. In designs with two or more levels of nesting, such as cluster randomized trials, several possible standard deviations could be used to compute standardized mean differences, so the concept of this effect-size measure is ambiguous. This chapter has two goals. One is to provide definitions of effect sizes that may be appropriate for cluster randomized trials. The second is to provide methods to estimate these effect sizes (and obtain standard errors for them) from statistics that are typically given in reports of research (that is, without a reanalysis of the raw data). To ease the exposition, I consider studies with two levels of nesting (for example, students within schools or individuals within communities) first, then consider effect sizes in studies with three levels of nesting (for example, students within classrooms within schools or individuals within plants within companies).