What We have Learned in 20 Years of IES Randomized Trials

Purpose and Contribution to Knowledge: Randomized trials are the gold standard for discovering causal effects, yet many randomized trials fail to find (statistically significant) effects of interventions. Critics would claim that if most trials fail to find effects for the interventions that they study, then most of the investment in trials leads to failed trials, nothing is learned from them and the investment is wasted. This project will explore what has been learned from the trials supported by IES, with particular emphasis on the trials that might be considered “failed trials.” Data: This project will examine all of the efficacy and effectiveness trials funded by IES and completed by 2020. It will seek to understand and document the direct and indirect impact of each project on practice, fundamental understanding of education, human capital for education research, and research methods. Data sources will include project abstracts from the IES website, study proposals, and reports of those studies, as well as from interviews with Principal Investigators. We will use internet searches to identify products (such as papers) and citations to the studies and web scraping to discover mentions of the studies in websites of state education agencies, school districts, regional education labs, and professional organizations. Data extracted from all of these sources will be systematically coded and coding reliability will be assessed systematically. Approach: We will document the products of each study (such as scientific papers and reports) as well as indications that they influenced practice either directly or indirectly. Scientific knowledge is not just created from single studies in isolation, but by studies building upon one another. We will build a database showing relations between each study and both prior and subsequent related studies. We will try to document the ways in which earlier studies influenced later studies through what they learned, though the personnel that they trained, and through the fundamental knowledge that they created. We will also try to document situations in which a study either created a methodological innovation that was later used in other studies or motivated other methodological research. Data Analysis: The data analysis will largely be descriptive. We will identify each study’s direct and indirect contributions to practice, to fundamental knowledge and understanding of education, contribution to human capital for education research, and impact on methodology (or lack thereof). We will produce and summarize two databases that identify the network structure of relations among the studies and the network structure of personnel who worked on the studies. Key Outcomes: The key outcomes of this project will include the data extracted from studies, the network databases of relations among studies and their personnel, and summaries of these databases. We will also include a set of case studies designed to illustrate the ways in which particular studies have made contributions to knowledge about education.