Making School Reform Work: New Partnerships for Real Change

Chapter 8. Rethinking Data Capacity

83 EIGHT Rethinking Data Capacity marguerite roza most urban cities lack the strategic information to successfully identify and implement a district reform strategy. Although the term “data-driven decisionmaking” has become a popular idea in school reform, urban districts do not have access to the right data to make the best decisions. That is not to say that districts do not have and use data, as is evidenced by the three-inch-thick binders of data handed to school board members at each meeting. The most commonly reported data describe the current conditions of schools, which often include average scores broken down by subject area, socioeconomic status, minority group, poverty, education level of students’ mothers, and so on. Information might also include attendance and dropout rates, disciplinary actions, and reams of budget documents that list nearly every line item expenditure recorded by the central office. Recently, states and districts have added accountability data equipped with ratings that compare individual schools’ scores with other schools with similar demographics. While voluminous and often important, these data do not inform leaders faced with having to make strategic decisions to improve their system. Rather than more descriptive data or even accountability data, districts now need to build a local analysis capacity that enables them to collect and utilize data for strategic planning and monitoring. As this chapter demonstrates, such a capacity needs to: —Inform the strategic decisions that affect students; 84 marguerite roza —Track the availability of key resources and their connection to student outcomes; —Provide early warnings of improvement or decline, school by school; and —Provide a districtwide capacity profile that enables leaders to focus their strategy. This chapter highlights these four key features of a local analysis capacity , discusses some of the challenges that districts meet in trying to gain this information, and demonstrates how and why an independent institution would be most suited to meet this need. Inform the Strategic Decisions That Affect Students Reform is now becoming high stakes. District leaders no longer have the luxury of continuing with business as usual or tweaking existing programs to make minor changes. Many urban superintendents and principals must show real results in the short term. But what are their options, and how do they know for sure what decisions will affect the bottom line—student performance? Paradoxically, current leaders get little help from the wave of performance data created as part of accountability programs or from the school and district report cards that are supposed to be used for this purpose. Many districts and thirty-six states collect data for “school report cards,” although these report cards generally do not provide the kind of information that leaders can use to improve their schools. One researcher, Russell French, has examined the data in these report cards: “One of the things that struck us most was that so many of the things that are reported have so little to do with student outcomes.”1 In his research, he found that, in many cases, what is reported is simply what is available or what is required by law, and that there is very little information that would offer insights into the factors that could contribute to varying performance levels.2 And while accountability data that profile performance by school, grade, and subject do highlight successes and trouble areas, such data do not clarify choices among options for leaders. “I need data that tell me what to do,” pleads one superintendent at a data workshop. The more relevant data this superintendent seeks reveal the links between feasible actions for leaders and the real results for students. An example rethinking data capacity 85 comes from an urban district in New York. Here a consultant was able to compare student performance in several different instructional programs designed for students with a specific type of disability (as shown in figure 81 ). The three programs, labeled CT, RD, and CTS, had evolved in different locations throughout the district and had never been examined for their impact on student performance. With this type of data, the district was able to compare the impact each program was having on students with the programs’ costs and feasibility . The district could then make a well-informed strategic decision to expand the CT and RD programs while eliminating CS. In contrast, most leaders are faced with data that compare students’ outcomes with factors over which school leaders have no control, such as the...