Can the Federal Government Improve Education Research?

Recent dissatisfaction with public education in the United States has been matched by dismay with the current state of education research. A common complaint is that education research is good at description and hypothesis generation but not at answering causal questions about the effects of education policies on student outcomes.¹ In this vein, many policymakers have expressed frustration that, as Ellen Condliffe Lagemann has noted, "education research has not yielded dramatic improvements in practice of the kind one can point to in medicine."² Such dissatisfaction has contributed to a number of recent federal policy changes intended to improve the quality of research in education, including the creation of a new Institute of Education Sciences (IES) to support increased experimentation within education and an emphasis on the use of teaching methods supported by "scientifically-based research" in the 2001 No Child Left Behind Act (NCLB).

In this paper we consider the possible effects of these recent changes on the state of education research. We focus on what might be termed program or policy evaluation—research that aims to support causal inferences about the efficacy of specific educational programs or policies. Examples include studies that examine whether smaller class size improves student achievement, whether a particular reading curriculum leads to increased reading comprehension, and whether "pull-out" programs are more effective than "push-in" programs for students with learning disabilities. It is important to note that a great deal of research in education does not aim to answer these types of questions but rather addresses the processes underlying teaching and learning or human development more generally. Although obviously this "basic" research can provide insights that directly benefit the development of more effective curriculum and pedagogy, our interest here is on the type of evaluation research designed to inform larger policy decisions relevant to education reform (though for convenience we sometimes refer generically to "education research").³

Understanding how recent policy changes may affect the state of education policy research and, ultimately, education practice requires some consideration of why the quality of past research has, on average, been so poor.⁴ Over the past several years, growing attention has been devoted to the question of why randomized field trials, considered the gold standard for causal inference in medical research and most of the social sciences, are not used more often in education. Candidate explanations include the practical and political difficulties associated with launching experiments in education and problems confronting the federal agencies that support education research.⁵ Left unanswered are several additional questions that are relevant in considering the effects of recent policy changes on education research. Why is the average quality of nonexperimental evaluation research in education so low? Why is much of the education evaluation work that is supported by private foundations and other nongovernmental sources of such low quality? More generally, why has "good" evidence from randomized experiments (or credible natural or quasi experiments) not driven out "bad" evidence over time?

In what follows, we suggest that the supply of high-quality education policy research is limited because the demand for it is limited. The supply of high-quality research is likely to be greatest in markets in which there is significant demand for the end product itself and good research can be distinguished from bad. Neither of these conditions currently seems to hold with much force in education because of "market failure" in the markets for both educational outcomes and education research.

In the past, there has been little demand for educational outcomes—specifically, academic achievement—among parents, educators, and policymakers. Possible explanations for this state of affairs include the difficulty parents have in determining the value added of their child's school, parental preferences for nonacademic as well as academic outcomes from schools, and a traditional lack of accountability within the public schools, which reduces the incentives facing educators to demand high achievement from their students. Insofar...