Comment

The validity of research depends to a large extent on the validity of the assumptions on which it is based. Some of the assumptions in William H. Schmidt's paper have been tested and found to be accurate. Others have been tested and found to be questionable. For others, little evidence exists one way or the other. Examining the assumptions underlying the paper is useful, therefore, as a basis for interpreting the findings and assessing their implications for public policy. These assumptions also have general relevance because they appear frequently in the education research literature.

I begin with basic assumptions that have been tested and found to be valid. First, students who have studied the material covered by a test will get higher test scores than those who have not. It is difficult to do well on a calculus test if you have never studied calculus. Second, the paper assumes a selection bias—that is, some students are advised, inappropriately, to take less demanding courses. That advice, in turn, reduces the students' potential for high academic achievement and closes options they otherwise might have had. The important point is that schools have a responsibility to offer each child the strongest possible educational experience.

From these well-tested assumptions, the paper moves to a set of assumptions for which there is less support. First, the paper assumes that the Third International Mathematics and Science Study (TIMSS) provides the information needed to draw conclusions about the quality of education in participating countries. The main problem is that Schmidt uses TIMSS data as the basis for his analysis.³⁵ In my view, TIMSS is so flawed that implications cannot be drawn about the quality of education in any of the participating countries.³⁶ Rankings of countries in TIMSS are based on simple comparisons of test scores in the final year of secondary school, without any controls for the large differences between participating countries on a wide range of variables. Therefore, it is impossible to learn from the study how variables such as the following affected student selectivity and, in turn, test score rankings: participation and exclusion rates of both schools and students being tested; percent of the age cohort who dropped out of school and therefore did not take the test; percent of students taking advanced assessments; average age and grade of students taking the test; special concentrated programs for different students; practices with respect to the inclusion or exclusion of low-achieving students, language minority students, students with disabilities, apprenticeship programs, and entire regions of the country in the test comparisons; the mix of public and private schools, comprehensive and specialized schools, and academic and vocational schools; tracking and coaching practices; family socioeconomic status (SES); and the consistency between the education program and the test.

Each of these variables can be expected to play a significant role in the extent to which the students taking the test represent a highly select group, not the general student population. The TIMSS study did not conduct a multivariate analysis to provide information about the contribution of each variable to the test score rankings. The variables are so confounded that how any of them, individually or in combination, affected the test scores cannot be determined. The use of these data, therefore, does not contribute either to research knowledge or to informed public policy.

The difficulty of unraveling the TIMSS findings is illustrated by tables 1-3, which show the wide differences between countries on several of the major variables. The tables, which are adapted from data presented in the TIMSS report, also explicitly show that few of the participating countries met the international sampling and other guidelines set forth by the TIMSS researchers.

Schmidt uses TIMSS data as the main basis for his findings and recommendations with respect to curriculum. Unfortunately, the study provides no guidance about curriculum or any other component of school systems. It serves, instead, primarily as a Rorschach test that reflects...