Judging Statistical Criticism

Bross (1960) proposes rules for statistical criticism, chiefly that critics bear the responsibility of proving the tenability of a counterhypothesis. This Comment makes three points. First, the higher the tenability standard, the more statisticians will be drawn into the local ground rules of a substantive field. Bross feared this prospect, yet his work exemplifies it. Second, more content needs to be given to the tenability standard across domains. Proving tenability may be untenable, for instance, when data is unavailable. Third, Bross’s proposal ultimately led him to espouse a quasi-judicial “adversary science” proceeding to resolve controversial issues of public policy (Bross, 1980). But Bross’s own involvement in a pilot at the Nuclear Regulatory Commission illustrates the difficulties with a “science court” model, with adversarialism potentially exacerbating rather than muting political conflict. I illustrate these points with the common setting of statistical evidence in an antidiscrimination suit, using data from the University of Texas at Austin School of Law. Ultimately, Bross’s work raises profound questions about the institutions for judging statistical criticism.