Comment by Carl F. Kaestle
[The Federal Role in Educational Research and Development]
Maris A. Vinovskis and I were colleagues at the University of Wisconsin many years ago, and we collaborated on a book and some articles back in the 1970s. We have different political instincts but agree on many things about the history of American education and about educational research. We have also had some similar relationships to the Office of [End Page 380] Educational Research and Improvement (OERI), the subject of Vinovskis's paper. We both have written about the history of the agency, and we both have spent some time around it. But Vinovskis on both of these counts--knowing the history of the agency and hanging around its hallways--has gone far beyond anything I have done, so my remarks on his paper are given with some modesty and as a friend. Still, worrying about the fate of the National Institute of Education (NIE) and OERI has been a Division I sport since 1972, so I am not unusual in having opinions on the matter.
Vinovskis begins his paper with an argument that policymakers need better federally sponsored research to evaluate federal programs designed to help disadvantaged youth and to create alternative approaches to educating these kids. This policy emphasis is understandable, not only because Vinovskis has been steeped in the literature on these programs for the past few years, but also because the education of disadvantaged children is an important and legitimate focus of federally funded research.
However, pinnng the justification for better federally funded education research principally on its potential usefulness in assisting federal education policy may be unrealistic. This may overestimate its potential, on the one hand, and give too narrow a view of the functions and audiences of federally funded education research, on the other. I do not think that solid educational research evidence will ever be a litmus test for the establishment of new federal policies, and I think its application to ongoing programs will always be controversial, even with more and better evaluation research. Nonetheless, the aim should be for more and better evaluation. To do so will require policymakers and researchers to collaborate in building a new cohort of better trained researchers, making strong provisions for evaluation in policies and legislation, and perhaps experimenting with ideas such as that proposed by Chester E. Finn Jr. to run state trials followed by tough evaluations, in advance of launching national policies in a given area.
Nonetheless, policymakers are not likely to achieve consensus about the effectiveness of education programs in advance of their launch or in their first few years of operation. Politicians have to decide whether to establish programs on the basis of necessarily fragmentary, preliminary evidence, often from analogous programs that are not the same or from pilot programs, the results of which can be argued either way. Whether the program is about compensatory education to fight poverty, inclusion [End Page 381] in special education, block grants for more responsive decisionmaking, systemic reform for higher academic achievement, or vouchers for generally more effective schooling, the problems are the same. A tidy, attractive model sometimes invoked from research and development (R&D) in business or the military begins with laboratory-scale production followed by evaluation, then moves to pilot-scale production and evaluation, and finally to larger-scale production and evaluation. However, this model will probably not save educational research and development. It does not always work well in other sectors, and it rarely pertains to education (although I would not mind seeing it attempted more often and more rigorously). The variables in education are not as controllable and the process is more complicated. Because the activity is important, public, and political, education R&D is not allowed the insulation and time that careful, research-based development and evaluation enjoys in other more protected spheres.
Policymakers should be given the best, toughest evaluations possible, as promptly as possible, and their limitations and usefulness should be made clear. But education researchers should not be expected to make quick summary judgments about complex educational processes and outcomes. They are better, in the short run, at assessing the importance of context, the ambiguity of program labels, the appearance and impact of unintended consequences, and other complexities. Over the longer haul, they may reach some consensus about trends in educational outcomes. Even when the drift of judgment about effectiveness seems to be going in one direction, there will always be diverse studies and results, and policymakers can select the research that supports their own policy instincts and interests.
Major compensatory programs are always moving targets. Good policymakers are interested not only in short-term learning measures but also in long-term outcomes, the persistence of academic gains, and nonacademic goals in late youth or early adulthood. Thus, the research has to be long term. In the meantime, while multiyear longitudinal research is progressing, the program is changed, presumably for the better. So when results arrive from Ypsilanti, the Prospects study, or some other source, researchers are evaluating data from a program as it existed some years before. The results are relevant, but not conclusive.
Again I am not arguing that tough-minded evaluations of federally funded education programs are not needed but that the launching and the [End Page 382] early adjustment of programs will often have to continue without thorough, credible evaluation of outcomes--whether it is big federal programs, Chicago decentralization, Milwaukee choice, or Success for All. It is the nature of the beast. A program's effectiveness cannot be fully evaluated until a substantial investment has been made in it. Even then, if it is tinkered with along the way (as it should be), the evaluations will always be a little out-of-date.
If I seem to be more skeptical than Vinovskis about education research in the service of federal policy, I am also reminded of other, more diffuse purposes of education research. In educational federalism (a complicated American invention), the tilt in educational governance is toward the state and district levels. Only a small portion of policy and practice is determined at the federal level. But education research sponsored by the federal government can have an influence on policy and practice determined at the state level, by the district, in the school building, in the classroom, or for an individual student. Federally sponsored research can serve all of these actors, and agencies such as OERI must keep all of them in mind. The school finance research of the 1970s was largely relevant at the state level; the recent work on reading is relevant to states, school districts, and individual teachers, as is work on cognitively guided instruction in math or the work on domain-specific knowledge conducted at the Learning and Research Development Center in Pittsburgh. So, while I agree with Vinovskis that the main agency charged with conducting educational research should be doing a more impressive job evaluating large federal initiatives and exploring alternatives, I would keep in mind the many nonfederal uses of federally funded education research.
Vinovskis considers possible areas for improvement in the structure, personnel, and purposes of OERI. I will select only a few for discussion: changing the structure of OERI, restoring its capacity for high-quality research, and bringing focus to its fragmented research mission.
One recurring issue in the twenty-eight-year history of OERI has been the desire to insulate it from politics and give it stability through some structural design. The same thing was heard in 1999, from various quarters. Vinovskis seems properly skeptical about this, though he says that the performance of the agency is so problematic that thoughts about restructuring should be entertained. I believe that the quest for a haven from politics is quixotic. It did not work with NIE (witness the treatment on Capitol Hill of Tom Glennan, NIE's first director, or the Reagan [End Page 383] administration's summary dismissal of the independent NIE research board chaired by Harold "Doc" Howe). And after the National Center for Education Statistics (NCES) had developed a remarkable reputation for independence and impartiality, it was thrust overnight into the Clinton administration's education program as the main developer and advocate of the president's Voluntary National Test. Politics can scale any walls in Washington, especially if the walls were built with federal dollars.
The costs of restructuring are fearsome and should not be taken lightly. When people look back on the creation of the Department of Education, the creation of OERI, even the more modest restructuring of NIE under Patricia Graham, they uniformly tell tales of paralysis, deep decline of morale, and preoccupation with bureaucratic adjustments in jobs and functions that last months and months. The authors of such changes later expressed doubts that structural changes matter much. In the oral histories I did for the National Research Council's committee on the previous OERI reauthorization, the following remarks were made about reorganizations that had taken place from the mid-1970s to the mid-1980s: Virginia Richardson, head of research on teaching at NIE, said there was "more cost than benefit"; Sally Kilgore, director of the Office of Research, said it was an "incredible distraction"; Chester Finn, assistant secretary of OERI, said, "The more I've been here, the less I think that you cause change by moving boxes around"; and Ernest Boyer, commissioner of education, said, "Structure is almost totally inconsequential."72 Isn't that some sort of bipartisan consensus? Before politicians undertake to save OERI by abolishing it and starting over, recall that one never "starts over" in Washington bureaucracies. New structures will not change ongoing obligations, existing staff, and--more important--existing attitudes. What is wrong is something deeper than structure and harder to change.
I agree with Vinovskis's central emphasis on the gradual, regrettable dilution of the research orientation and research capacity of OERI, a woeful, vicious circle. I join him in applauding Kent McGuire's emphasis on repairing this capacity. I agree with his well-phrased statement that there is no easy or ideal answer. I agree that the main problems are research capacity, stability, leadership, and focus. Repairing these does not preclude some structural changes within the framework of OERI. The notion that something like the success of NCES could be replicated on the research side of OERI lingers. Some have urged a fixed term for the head [End Page 384] of OERI. I do not know what kind of animal a fixed-term assistant secretary is. It sounds like a mythical animal, like Doctor Doolittle's Push-me-Pull-you, with two heads. But perhaps there could be a commissioner of education research parallel to the commissioner of education statistics, in charge of a Center for Education Research.
These changes might facilitate a renewed dedication to quality research, with the assistant secretary still in charge of the office as a whole, with a reduced portfolio of improvement activities, plus two centers with fixed-term commissioners. The twin demand would be to reduce the fragmentation of the research agenda itself. Vinovskis rightly points out that this will necessitate hard thinking about the role of the labs and centers, and, I would add, the five institutes within OERI, which are diverse in their coverage and have not fulfilled their potential following that structural change within OERI. They were not funded as amply as some had hoped. Their directorships were too long filled with acting appointments, and then permanently with agency staff, not the visible outsiders who would bring new energy and prominence to the agency's research mission. Apart from these considerations, if OERI adopts the proposal of the National Education Research Planning and Priorities Board for a more focused research agenda, it will mean that the structure and activities of the institutes need to be reevaluated.
These commitments--to revitalize the central research mission of OERI and to focus its research agenda on a shorter list of priorities--might mean internal structural changes of some magnitude. But they would remain within the structure of OERI. Abolishing the agency to keep it from politics is, I think, wishful thinking. Abolishing it to escape its reputation and its diminished capacity avoids the issues that have led to its low reputation and its diminished capacity, issues that will not go away with a new acronym and a new address. What is needed is a new consensus; a consensus spanning Congress, the secretary of education's office, the agency itself, the research community, and the leaders of policy and practice groups; a consensus that OERI must define, sponsor, evaluate, synthesize, and use high-quality research around a focused agenda of long-term, practical importance. It would follow from such a consensus that OERI must recruit a small cadre of research leaders. It needs to have in place shortly after the new administration takes office an energetic assistant secretary with first-rate credentials in research leadership and the uses of research. It will have to redirect the existing staff to [End Page 385] a new priority on quality--quality peer review, quality monitoring, and new opportunities for professional development directly related to research. To survive, the agency will have to dedicate itself to the proposition that better work on a smaller agenda is mandatory.
The agency desperately needs to reverse the downward spiral of low performance, low respect, low expectation, and low resources. The agency should not be cast aside, and in my opinion, its salvation does not depend mainly upon restructuring. Instead, OERI needs a change of will, a sense of urgency that the work must be done better, and a newfound confidence that it can be done better. Perhaps this will not happen until (in the words of the 1960s rock song) "the moon is in the Seventh House and Jupiter aligns with Mars," but it is, nonetheless, what the agency needs.
