The Controversy over the National Assessment Governing Board Standards

This paper provides an analysis of the controversy surrounding the standard setting process conducted by ACT Inc. for the National Assessment Governing Board (NAGB). ¹ This process is the most thoroughly planned, carefully executed, exhaustively evaluated, completely documented, and most visible of any standard setting process of which I am aware. Extensive research was conducted to determine how best to develop each step in the process. ² A distinguished team of experts guided the process through its development and implementation. ³ And, the process has been open to scrutiny with evaluators observing the design and implementation of every step.

Any process can be improved with experience and with continuing research and development. Better methods for setting standards likely will be created in the future. Until such developments occur, however, this process--called the achievement levels setting (ALS) process by NAGB--is the model for how standard setting should be done. The question I attempt to answer here is: If the standard setting process is of such high quality, why are the standards set by the process so controversial?

Although I think extremely well of the NAGB standard setting process, interpreting the results of the ALS process is a very complex undertaking. A difference has become evident between the technical accuracy of the standards and the clarity of meaning for the standards that were set. The technical quality of the standards is very high. Statistical analyses have shown that the standards are well within the accepted bounds for amount of error in the estimated cutscores, and follow-up validity studies have provided supportive evidence for the technical quality of the standards. ⁴ Cutscores are points on the score scale used for reporting that separate levels of performance such as pass/fail or, in this case, the achievement level categories. Further, the standards are based on the reasoned judgments of a carefully selected group of qualified individuals. ⁵ These individuals reviewed the outcomes of their work and endorsed the resulting standards as being reasonable and representing their best efforts. ⁶

But as important as the technical and procedural quality of the standards is, the clarity of information communicated by the standards-based reporting is at least as important. As the Committee on the Evaluation of National and State Assessments of Educational Progress concluded: "Standards-based reporting is intended to be useful in communicating student results to the public and policy makers. However, sufficient research is not yet available to determine how various audiences interpret and use NAEP's [National Assessment of Educational Progress's] achievement-level results." ⁷

NAGB reports three standards of performance for many content areas included in the National Assessment of Educational Progress. NAGB calls these standards achievement levels (ALs). NAGB policy defines three ALs with these labels: Basic, Proficient, and Advanced. For content areas with ALs, NAEP reports the numerical value of the cutscores that delimit the three ALs on the NAEP score scale and the percentage of students in tested grade levels that are above each cutscore. Interpretive aids for the results include descriptions of skills and knowledge for students estimated to be in ranges on the score scale defined by the ALs and examples of items with estimated levels of performance for each AL. ⁸ An AL is the range of the score scale from an AL boundary to the next higher AL. For example, the Basic level ranges from the lowest score for Basic to just below the start of Proficient. The Advanced level ranges from just above Proficient to an unspecified maximum possible level of performance. In practice, the Advanced level does not have an upper boundary. Three AL ranges correspond to the standards for Basic, Proficient, and Advanced. To be determined is whether the method for reporting NAEP results communicates useful information about the levels of educational achievement of students in the United States.

I am not a...