Adding Design Elements to Improve Time Series Designs: No Child Left Behind as an Example of Causal Pattern-Matching

Abstract:Some form of a short interrupted time series (ITS) is often used to evaluate state and national programs. An ITS design with a single treatment group assumes that the pretest functional form can be validly estimated and extrapolated into the postintervention period where it provides a valid counterfactual. This assumption is problematic. Ambiguous preintervention functional forms are common, as are other factors affecting posttest means and slopes. Using No Child Left Behind as an example, we demonstrate how adding multiple design elements to the basic ITS structure serves to promote causal inference by limiting alternative interpretations. No added design element is perfect by itself, but we argue that they collectively provide a strong causal warrant when the predictions they engender are complex, the results “cohere” with the predictions, and no alternative can fit the same pattern of predictions even if it can fit some of the