How large an effect can we expect from school reforms?

Background/Context: Determining the effectiveness of reform strategies is a major part of the current and future educational research agenda. Effects of education reforms will be evaluated largely quantitatively, and an important aspect of this work will be judging how well reform strategies work. The rhetoric of contemporary school reform suggests two somewhat different solutions to the problem of the interpretive frame. One solution is derived from the idea that the goal of school reform is to reduce the achievement gaps between minority groups such as Blacks or Hispanics and Whites, rich and poor, and males and females. The second solution is derived from a similar idea that school reforms are intended to reduce the achievement gap between lower and higher achieving schools. Purpose: The purpose of this paper is to explore these two alternative frameworks for interpreting the effects of school reforms and to gain insight about the implications of each of the frameworks for interpreting these effects. Research Design: We use NAEP trend data and examine empirical evidence about the implications of these two frameworks for judging the effects of school reforms. Our study is correlational and uses observational data from the 1970s, 1980s, and 1990s. Findings: We find that these two frameworks lead to different judgments about whether the effects of reforms are large enough to be important. We argue that the normative distribution of school effects framework is more appropriate than the other framework for interpret-ing the likely magnitude of school reform effects. In addition, we show that interpreting the magnitude of the effects of school reforms in terms of individual variation and the achievement gaps between important student groups may not only be disappointing, but also misleading. Conclusions/Recommendations: The results of this study can be used to provide a way to obtain plausible reform effects for designing studies of school reform. The results of this study can also provide a context in which to evaluate the results of studies of the effectiveness of school reform in terms of national data. Our study also illustrates one way in which survey data can contribute to evidence-based policy formation. The distribution of the observed school effects provides a basis for estimating plausible effect magnitudes far planning intervention studies. The analyses of school effects can also provide a context for interpreting treatment effects within the context of the observed variation. It permits the policy researcher to explain the implications of reform effects within the backdrop of observed variation within which any intervention will operate.