The Cult of Statistical Significance: How the Standard Error Costs Us Jobs, Justice, and Lives

Contents

Contents Preface xv Acknowledgments xix A Significant Problem  In many of the life and human sciences the existence/whether question of the philosophical disciplines has substituted for the size-matters/how-much question of the scientific disciplines . The substitution is causing a loss of jobs, justice, profits, environmental quality, and even life. The substitution we are worrying about here is called “statistical significance”— a qualitative, philosophical rule that has substituted for a quantitative, scientific magnitude and judgment. . Dieting “Significance” and the Case of Vioxx  Since R. A. Fisher (1890–1962) the sciences that have put statistical significance at their centers have misused it. They have lost interest in estimating and testing for the actual effects of drugs or fertilizers or economic policies. The big problem began when Fisher ignored the size-matters/how-much question central to a statistical test invented by William Sealy Gosset (1876–1937), so-called Student’s t. Fisher substituted for it a qualitative question concerning the “existence” of an effect, by which he meant “low sampling error by an arbitrary standard of variance.” Forgetting after Fisher what is known in statistics as a “minimax strategy,” or other “loss function,” many sciences have fallen into a sizeless stare. They seek sampling precision only. And they end by asserting that sampling precision just is oomph, magnitude, practical significance. The minke and sperm whales of Antarctica and the users and makers of Vioxx are some of the recent victims of this bizarre ritual. . The Sizeless Stare of Statistical Significance  Crossing frantically a busy street to save your child from certain death is a good gamble. Crossing frantically to get another mustard packet for your hot dog is not. The size of the potential loss if you don’t hurry to save your child is larger, most will agree, than the potential loss if you don’t get the mustard. But a majority of scientists in economics, medicine, and other statistical fields appear not to grasp the difference. If they have been trained in exclusively Fisherian methods (and nearly all of them have) they look only for a probability of success in the crossing—the existence of a probability of success better than .99 or .95 or .90, and this within the restricted frame of sampling—ignoring in any spiritual or financial currency the value of the prize and the expected cost of pursuing it. In the life and human sciences a majority of scientists look at the world with what we have dubbed “the sizeless stare of statistical significance.” . What the Sizeless Scientists Say in Defense  The sizeless scientists act as if they believe the size of an effect does not matter. In their hearts they do care about size, magnitude, oomph. But strangely they don’t measure it. They substitute “significance” measured in Fisher’s way. Then they take the substitution a step further by limiting their concern for error to errors in sampling only. And then they take it a step further still, reducing all errors in sampling to one kind of error—that of excessive skepticism, “Type I error.” Their main line of defense for this surprising and unscientific procedure is that, after all, “statistical significance,” which they have calculated, is “objective.” But so too are the digits in the New York City telephone directory, objective, and the spins of a roulette wheel. These are no more relevant to the task of finding out the sizes and properties of viruses or star clusters or investment rates of return than is statistical significance. In short, statistical scientists after Fisher neither test nor estimate, really, truly. They “testimate.” . Better Practice: ␤-Importance vs. ␣-“Significance”  The most popular test was invented, we’ve noted, by Gosset, better known by his pen name “Student,” a chemist and brewer at Guinness in Dublin. Gosset didn’t think his test was very important to his main goal, which was of course brewing a good beer at a good price. The test, Gosset warned right from the beginning, does not deal with substantive importance . It does not begin to measure what Gosset called “real error” and “pecuniary advantage ,” two terms worth reviving in current statistical practice. But Karl Pearson and especially the amazing Ronald Fisher didn’t listen. In two great books written and revised during the 1920s and 1930s, Fisher imposed a Rule of Two: if a result departs from an assumed hypothesis by two or more standard deviations of its own sampling variation, regardless of the size of the prize and...