In lieu of an abstract, here is a brief excerpt of the content:

Brookings Papers on Education Policy 2001 (2001) 181-229



[Access article in PDF]

Searching for Indirect Evidence for the Effects of Statewide Reforms

David Grissmer and Ann Flanagan

[Comment by Philip Uri Treisman and Edward J. Fuller]
[Comment by Robert H. Meyer]
[Tables]

States are the primary policymakers in several important areas of K-12 education. States, on average, provide approximately one-half of educational funding to school districts. Thus they are instrumental in determining how much is spent on education and how that money is used to reduce inequity in funding among school districts. States also are instrumental in determining who teaches, what is taught, and the conditions for teaching. These state policies include setting teacher certification standards, establishing maximum class-size and minimum graduation requirements, setting educational standards in subjects, and establishing methods of assessing student performance and methods of accountability for teachers, schools, and school districts. States also influence the extent of early education through kindergarten and preschool regulations and, in some states, through subsidizing prekindergarten for lower income families.

Given their dominant role in educational funding and regulation, states not surprisingly have been the primary initiators of the latest wave of educational reform starting in the mid-1980s. 1 Perhaps the most widespread initiative is a systemic reform movement that includes defining educational standards, aligning curriculum and teacher professional development to the standards, and having some form of assessment and accountability with respect to the standards. 2 While simple in concept, design and implementation are arduous, and states have made varying amounts of progress. 3 Many states have also passed legislation authorizing charter schools, school choice, or contract schools based on the assumption that public schools are unreformable without external competition and parental choice. [End Page 181]

Having fifty states take different approaches to education can provide a long-term advantage if research and evaluation can identify successful and unsuccessful approaches. If states adopted fairly uniform policies and implemented similar reform initiatives at the same time, then evaluation would be difficult. However, the states have a wide degree of variation in their educational policies and practices, making between-state variation a significant part of total variation. For instance, two-thirds of the variance in district per pupil expenditures is between, not within, states. 4 Reform initiatives across states have also varied widely both in substance and timing of implementation, making states a potentially valuable source of evidence about the effects of resources and reforms.

Analyzing Comparative State Achievement

Until 1990, no statistically valid measures existed of the achievement of representative samples of students across states. 5 Comparative state analysis became possible when the National Assessment of Educational Progress (NAEP) tests were administered to representative samples of students across a voluntary sample of states in 1990, 1992, 1994, 1996, and 1998. Nine tests were given in reading and mathematics at either the fourth or eighth grade. Each test was given to approximately twenty-five hundred students, with forty-nine states administering at least one test. These tests represent the first valid, comparable measures of achievement of representative samples of children in various states. These data are unique in that they make it possible to assess comparative achievement across states.

While these tests present an opportunity to assess comparative state achievement performance, significant analytical and methodological problems arise in obtaining the kind of reliable results needed by policymakers. First, previous research would suggest that family variables would account for a substantial part of the variation of scores across states because of wide variation in their demographic composition and family characteristics. The family variables collected with NAEP are limited, and those collected are student-reported by fourth- and eighth-grade students, making their quality poor. Without accurate family control variables, the results from any analysis will be problematic. Any analysis needs to address this issue.

Second, the current methodological debate concerns whether aggregate- or individual-level analysis yields more accurate results. 6 The NAEP data can [End Page 182] be analyzed at the individual level to yield statewide parameters or at the aggregate state level...

pdf

Share