-
Appendix A: Methodology for Comparing U.S. and International Performance
- Brookings Institution Press
- Chapter
- Additional Information
This appendix describes the strategy for equating scores across states and nations, first, in terms of proficient and advanced levels of performance (reported in chapters 3 and 4), and, second, in terms of changes in performance over time (reported in chapter 6). Comparing Proficient and Advanced Levels across States and Countries For the analyses in chapters 3 and 4, the goal is to compare how students in the different states in the United States are doing with respect to their peers in terms of reaching proficient and advanced levels, respectively. We want to do so with as much detail (by state and social group) as the data permit. To obtain this information, we build a crosswalk between the National Assessment of Educational Progress (NAEP) and the Program for International Student Assessment (PISA), which was administered to representative samples of fifteen-year-old students in sixty-eight of Appendix A MethodoLogy for CoMPAring U.S. And internAtionAL PerforMAnCe 105 13291-08_AppA_3rdPgs.indd 105 6/6/13 10:47 AM 106 Appendix A the world’s school systems. Note that the sixty-eight school systems include the entirety of sixty-five countries along with three subcountries included in PISA reports: Hong Kong, Macao, and Shanghai. Hong Kong and Macao operated as independent states for a long time and currently have special status. The crosswalk is developed by looking at the percentage of U.S. students who reach the proficient and advanced levels, respectively, on the NAEP assessment and at the equivalent cutoff score in PISA for that percentage of U.S. students. This gives us the equivalent of the PISA thresholds, allowing us to estimate comparable rates of students performing at the proficient and advanced levels, respectively, for all countries and to compare student performance in each of the states in the United States with that of their international peers. Our analysis relies on test score information from young adults collected by NAEP in 2007 and PISA in 2009.1 NAEP is a large, nationally representative assessment of student performance that has been administered periodically since the late 1960s to U.S. students in fourth and eighth grades and at the age of seventeen. Since 2003 it has provided achievement data for students in each of the fifty states in mathematics and reading. PISA is an internationally standardized assessment of student performance in mathematics , science, and reading established by the Organization for Economic Cooperation and Development (OECD). It was administered in 2000, 2003, 2006, and 2009 to representative samples of fifteen-year-olds in all OECD countries as well as in many others.2 NAEP is governed by the National Assessment Governing Board (NAGB), which consists of twenty-six educators and other public figures appointed by the U.S. secretary of education. We rely on the 2007 samples of NAEP for eighth-grade public and private school students in each of the fifty states. For each of these jurisdictions, NAEP 2007 calculates the percentage of students who perform at three levels: basic, proficient, and advanced; our analyses in chapters 3 and 4, respectively, use the latter two performance levels. 13291-08_AppA_3rdPgs.indd 106 6/6/13 10:47 AM [44.221.43.88] Project MUSE (2024-03-28 18:25 GMT) 107 Appendix A Our crosswalk from NAEP to PISA aims to identify the relative performance of the class of 2011. NAEP examinations are given to eighth graders, January through March, when most students are thirteen to fourteen years of age. PISA examinations are given to a random sample of students at the age of fifteen, the age at which approximately 70 percent of U.S. students are in tenth grade.3 To construct the achievement comparisons for the class of 2011, we rely upon the 2007 NAEP test and the 2009 PISA test. In comparing the performance of the class of 2011 on the NAEP and PISA tests at these two different points in time, we assume that no event happened between eighth and tenth grade that significantly altered the performance of American students relative to that of students in other countries. Because U.S. students took both the NAEP and the PISA, it is possible to find the score on the PISA that is tantamount to scoring at a specific performance level on the NAEP, that is, the score that will yield the same percentage of U.S. students as scored at this level on the NAEP. We describe this crosswalk exercise for the example of performance at the proficient...