The Cult of Statistical Significance
How the Standard Error Costs Us Jobs, Justice, and Lives
Publication Year: 2008
Published by: University of Michigan Press
The implied reader of our book is a significance tester, the keeper of numerical things. We want to persuade you of one claim: that William Sealy Gosset (1876–1937)—aka “Student” of Student’s t-test—was right and that his difficult friend, Ronald A. Fisher, though a genius, was wrong...
We thank above all Morris Altman, who organized a session at the American Economic Association meetings in San Diego on these matters (January 2004) and then edited the articles into a special issue of the...
A Significant Problem
For the past eighty years it appears that some of the sciences have made a mistake by basing decisions on statistical “significance.” Although it looks at first like a matter of minor statistical detail, it is not. Statistics, magnitudes, coefficients are essential scientific tools. No one...
1. Dieting “Significance” and the Case of Vioxx
Suppose you want to help your mother lose weight and are considering two diet pills with identical prices and side effects. You are determined to choose one of the two pills for her. The first pill, named Oomph, will on average take off twenty pounds...
2. The Sizeless Stare of Statistical Significance
The very word significance, emphasizing its statistical meaning, is often used prominently in an advertisement, especially for pharmaceuticals. Xanax is a product of the Upjohn Company designed to alleviate clinical anxiety. In May 1983 Upjohn ran a two-page spread...
3. What the Sizeless Scientists Say in Defense
You will be uneasy. You will say again, “How can this be?” Statisticians and statistical scientists sometimes think they disagree with our strictures on null-hypothesis significance testing (Hoover and Siegler 2008; Mc- Closkey and Ziliak 2008). They are made unhappy by our radical...
4. Better Practice: beta-Importance vs. alpha-“Significance”
We are not by any means the first people, even in economics, to express such indignation (Edgeworth 1885, 215; Morgenstern 1950, 92–93; Savage 1954, 1971a; Arrow 1959; Tullock 1959; De Finetti 1971; Leamer 1978; Mayer 1979; Zellner 1984)...
5. A Lot Can Go Wrong in the Use of Significance Tests in Economics
We are quantitative economists. Our scientific work has concerned British industrial history and medieval agriculture, labor market statistics and the history of American charity, all viewed quantitatively. It has asked, How Much...
6. A Lot Did Go Wrong in the American Economic Review during the 1980s
Table 6.1 shows the results for the 1980s, arranged from worst result to best by question. That is, • Seventy percent of the articles in the...
7. Is Economic Practice Improving?
Our 1996 article reporting these findings, we repeat, had almost no impact. Horowitz and a few others have told us that they disagree. They see progress and give us credit for causing some of it. That would be nice. But lack of impact is typical in other fields subjected to such criticism, for...
8. How Big Is Big in Economics?
Of course, not everyone gets everything wrong. The American Economic Review over the past two decades has been filled with superb economic science. In our opinion a good fraction of all the articles we read can be described this way—even though a supermajority make significant...
9. What the Sizeless Stare Costs, Economically Speaking
You cannot “test” mechanically for nonzero along some scale that has no dimension of substance and cost. How many molecules do you suppose you share with William Shakespeare? We mean molecules in your body that were once in his? Surprisingly, the correct answer, in view...
10. How Economics Stays That Way: The Textbooks and the Referees
The proximate cause of the unhappy situation in economics is that almost all the teachers of econometrics claim that statistical significance is the same thing as scientific significance. The econometrician David Hendry, for example, is famous for saying “test, test, test,” where the phrase means...
11. The Not-Boring Rise of Significance in Psychology
Economists, we have noted repeatedly, are not the only scientists to fall short of real significance. Psychologists have done so for many decades now. An addiction to transforms of categorical data, a dependence on absolute criteria of Type I error, and a fetish for asterisk psychometrics have...
12. Psychometrics Lacks Power
The cost of the psychological addiction to statistical significance can be measured by the “power function.” Power asks, “What in the proffered experiment is the probability of correctly rejecting the null hypothesis, concluding that the null hypothesis is indeed false when it...
13. The Psychology of Psychological Significance Testing
Why have psychologists been unwilling to listen? One reason seems to be insecurity in a so-called soft or subjective field. Recall even the learned Paul Meehl, a psychological scientist as well as a philosopher, speaking of his own field as “soft.” The “hard/soft” dichotomy is surely a poor one...
14. Medicine Seeks a Magic Pill
In general you wish to know the probability that your medical hypothesis, H, is true in view of the incomplete facts of the world. This is a problem of inference, inferring the likelihood of a result from data. If the symptoms of cholera start in the digestive system, then ingestion of something...
15. Rothman’s Revolt
As early as 1978 the situation was sufficiently dire that two contributors to the New England Journal of Medicine, Drummond Rennie and Kenneth J. Rothman, published op-ed pieces in the journal pages about the matter (Rennie 1978; Rothman 1978). Rennie, the deputy editor of the...
16. On Drugs, Disability, and Death
By 1988 the International Committee of Medical Journal Editors had been sufficiently pressured by the Rothmans and Altmans to revise their “uniform requirements for manuscripts submitted to biomedical journals.” “When possible,” the committee wrote, “quantify findings and present...
17. Edgeworth’s Significance
How could such a strange scientific turn have taken place? How could it be that so many sciences became sizeless and (p < .05) hazardous to health and wealth? Francis Ysidro Edgeworth (1845–1926), the inventor of the very word...
18. “Take 3sigma as Definitely Significant”: Pearson’s Rule
The argument from odds is old. Stephen Stigler notes that one might read a significance test into the clinical trial in the Book of Daniel (1:12–16), though “[i]t may be a stretch” as “the significance level and even the test statistic were left vague there...
19. Who Sits on the Egg of Cuculus Canorus? Not Karl Pearson
An early case, applied to the eggs of the cuckoo bird, illustrates literally the feel of substantive as against statistical significance. “The Egg of Cuculus Canorus” (1901) by Oswald Latter was one of the first articles published in...
20. Gosset: The Fable of the Bee
Karl Pearson couldn’t learn from Gosset. But Gosset couldn’t learn much from Pearson, either. “I am bound to say that I did not learn very much from his lectures,” he told Egon. “I never did from anyone’s and my mathematics were inadequate for the task,” he continued...
21. Fisher: The Fable of the Wasp
If Gosset was the Bee, his difficult friend Fisher was the Wasp. Gosset patiently tried for a quarter century to teach Fisher about human relations, such as the importance of being kind and telling the truth and practicing humility and giving credit to other scientists and being accurate about...
22. How the Wasp Stung the Bee and Took over Some Sciences
In 1922 Fisher wrote to Gosset, soliciting Gosset’s new table of t. Fisher was eager to put the updated table in the first edition of his Statistical Methods for Research Workers. They would call Gosset’s tables “the table of ‘Student’s’...
23. Eighty Years of Trained Incapacity: How Such a Thing Could Happen
Thorstein Veblen—Oswald Veblen’s uncle, we noted, and so, astonishingly, the intellectual granduncle of Harold Hotelling—spoke in a famous phrase of the “trained incapacity” of a businessman to attend to anything but pecuniary profit. In fact Veblen used the phrase only once, on page...
24. What to Do
Our first suggestion is mild and educational. Scientists in the sizeless sciences need to start telling each other to seek substance. They need to stop believing that the translation of a problem into probability space relieves them of the need to consider oomph and loss functions. The ritualism of...
A Reader’s Guide
Although our approach is mostly nontechnical, we assume the reader is broadly familiar with testing, estimation, and error statistics as used in the life and human sciences. Some readers may appreciate guidance along these lines...
Page Count: 352
Publication Year: 2008
Series Title: Economics, Cognition, and Society
See more Books in this Series
MUSE Marc Record: Download for The Cult of Statistical Significance