The Cult of Statistical Significance

Contents

Download PDF (144.3 KB)
pp. ix-xiv

Preface

Download PDF (110.7 KB)
pp. xv-xviii

The implied reader of our book is a significance tester, the keeper of numerical things. We want to persuade you of one claim: that William Sealy Gosset (1876–1937)—aka “Student” of Student’s t-test—was right and that his difficult friend, Ronald A. Fisher, though a genius, was wrong...

Acknowledgments

Download PDF (133.6 KB)
pp. xix-xxiii

We thank above all Morris Altman, who organized a session at the American Economic Association meetings in San Diego on these matters (January 2004) and then edited the articles into a special issue of the...

A Significant Problem

Download PDF (188.7 KB)
pp. 1-22

For the past eighty years it appears that some of the sciences have made a mistake by basing decisions on statistical “significance.” Although it looks at first like a matter of minor statistical detail, it is not. Statistics, magnitudes, coefficients are essential scientific tools. No one...

1. Dieting “Significance” and the Case of Vioxx

Download PDF (247.4 KB)
pp. 23-32

Suppose you want to help your mother lose weight and are considering two diet pills with identical prices and side effects. You are determined to choose one of the two pills for her. The first pill, named Oomph, will on average take off twenty pounds...

2. The Sizeless Stare of Statistical Significance

Download PDF (175.5 KB)
pp. 33-41

The very word significance, emphasizing its statistical meaning, is often used prominently in an advertisement, especially for pharmaceuticals. Xanax is a product of the Upjohn Company designed to alleviate clinical anxiety. In May 1983 Upjohn ran a two-page spread...

3. What the Sizeless Scientists Say in Defense

Download PDF (211.7 KB)
pp. 42-56

You will be uneasy. You will say again, “How can this be?” Statisticians and statistical scientists sometimes think they disagree with our strictures on null-hypothesis significance testing (Hoover and Siegler 2008; Mc- Closkey and Ziliak 2008). They are made unhappy by our radical...

4. Better Practice: beta-Importance vs. alpha-“Significance”

Download PDF (165.2 KB)
pp. 57-61

We are not by any means the first people, even in economics, to express such indignation (Edgeworth 1885, 215; Morgenstern 1950, 92–93; Savage 1954, 1971a; Arrow 1959; Tullock 1959; De Finetti 1971; Leamer 1978; Mayer 1979; Zellner 1984)...

5. A Lot Can Go Wrong in the Use of Significance Tests in Economics

Download PDF (191.5 KB)
pp. 62-73

We are quantitative economists. Our scientific work has concerned British industrial history and medieval agriculture, labor market statistics and the history of American charity, all viewed quantitatively. It has asked, How Much...

6. A Lot Did Go Wrong in the American Economic Review during the 1980s

Download PDF (203.0 KB)
pp. 74-78

Table 6.1 shows the results for the 1980s, arranged from worst result to best by question. That is, • Seventy percent of the articles in the...

7. Is Economic Practice Improving?

Download PDF (204.8 KB)
pp. 79-88

Our 1996 article reporting these findings, we repeat, had almost no impact. Horowitz and a few others have told us that they disagree. They see progress and give us credit for causing some of it. That would be nice. But lack of impact is typical in other fields subjected to such criticism, for...

8. How Big Is Big in Economics?

Download PDF (190.2 KB)
pp. 89-97

Of course, not everyone gets everything wrong. The American Economic Review over the past two decades has been filled with superb economic science. In our opinion a good fraction of all the articles we read can be described this way—even though a supermajority make significant...

9. What the Sizeless Stare Costs, Economically Speaking

Download PDF (170.9 KB)
pp. 98-105

You cannot “test” mechanically for nonzero along some scale that has no dimension of substance and cost. How many molecules do you suppose you share with William Shakespeare? We mean molecules in your body that were once in his? Surprisingly, the correct answer, in view...

10. How Economics Stays That Way: The Textbooks and the Referees

Download PDF (194.5 KB)
pp. 106-122

The proximate cause of the unhappy situation in economics is that almost all the teachers of econometrics claim that statistical significance is the same thing as scientific significance. The econometrician David Hendry, for example, is famous for saying “test, test, test,” where the phrase means...

11. The Not-Boring Rise of Significance in Psychology

Download PDF (180.3 KB)
pp. 123-130

Economists, we have noted repeatedly, are not the only scientists to fall short of real significance. Psychologists have done so for many decades now. An addiction to transforms of categorical data, a dependence on absolute criteria of Type I error, and a fetish for asterisk psychometrics have...

12. Psychometrics Lacks Power

Download PDF (219.1 KB)
pp. 131-139

The cost of the psychological addiction to statistical significance can be measured by the “power function.” Power asks, “What in the proffered experiment is the probability of correctly rejecting the null hypothesis, concluding that the null hypothesis is indeed false when it...

13. The Psychology of Psychological Significance Testing

Download PDF (194.7 KB)
pp. 140-153

Why have psychologists been unwilling to listen? One reason seems to be insecurity in a so-called soft or subjective field. Recall even the learned Paul Meehl, a psychological scientist as well as a philosopher, speaking of his own field as “soft.” The “hard/soft” dichotomy is surely a poor one...

14. Medicine Seeks a Magic Pill

Download PDF (239.5 KB)
pp. 154-164

In general you wish to know the probability that your medical hypothesis, H, is true in view of the incomplete facts of the world. This is a problem of inference, inferring the likelihood of a result from data. If the symptoms of cholera start in the digestive system, then ingestion of something...

15. Rothman’s Revolt

Download PDF (178.1 KB)
pp. 165-175

As early as 1978 the situation was sufficiently dire that two contributors to the New England Journal of Medicine, Drummond Rennie and Kenneth J. Rothman, published op-ed pieces in the journal pages about the matter (Rennie 1978; Rothman 1978). Rennie, the deputy editor of the...

16. On Drugs, Disability, and Death

Download PDF (276.7 KB)
pp. 176-186

By 1988 the International Committee of Medical Journal Editors had been sufficiently pressured by the Rothmans and Altmans to revise their “uniform requirements for manuscripts submitted to biomedical journals.” “When possible,” the committee wrote, “quantify findings and present...

17. Edgeworth’s Significance

Download PDF (202.8 KB)
pp. 187-192

How could such a strange scientific turn have taken place? How could it be that so many sciences became sizeless and (p < .05) hazardous to health and wealth? Francis Ysidro Edgeworth (1845–1926), the inventor of the very word...

18. “Take 3sigma as Definitely Significant”: Pearson’s Rule

Download PDF (158.2 KB)
pp. 193-202

The argument from odds is old. Stephen Stigler notes that one might read a significance test into the clinical trial in the Book of Daniel (1:12–16), though “[i]t may be a stretch” as “the significance level and even the test statistic were left vague there...

19. Who Sits on the Egg of Cuculus Canorus? Not Karl Pearson

Download PDF (144.0 KB)
pp. 203-206

An early case, applied to the eggs of the cuckoo bird, illustrates literally the feel of substantive as against statistical significance. “The Egg of Cuculus Canorus” (1901) by Oswald Latter was one of the first articles published in...

20. Gosset: The Fable of the Bee

Download PDF (212.9 KB)
pp. 207-213

Karl Pearson couldn’t learn from Gosset. But Gosset couldn’t learn much from Pearson, either. “I am bound to say that I did not learn very much from his lectures,” he told Egon. “I never did from anyone’s and my mathematics were inadequate for the task,” he continued...

21. Fisher: The Fable of the Wasp

Download PDF (251.3 KB)
pp. 214-226

If Gosset was the Bee, his difficult friend Fisher was the Wasp. Gosset patiently tried for a quarter century to teach Fisher about human relations, such as the importance of being kind and telling the truth and practicing humility and giving credit to other scientists and being accurate about...

22. How the Wasp Stung the Bee and Took over Some Sciences

Download PDF (172.1 KB)
pp. 227-237

In 1922 Fisher wrote to Gosset, soliciting Gosset’s new table of t. Fisher was eager to put the updated table in the first edition of his Statistical Methods for Research Workers. They would call Gosset’s tables “the table of ‘Student’s’...

23. Eighty Years of Trained Incapacity: How Such a Thing Could Happen

Download PDF (159.2 KB)
pp. 238-244

Thorstein Veblen—Oswald Veblen’s uncle, we noted, and so, astonishingly, the intellectual granduncle of Harold Hotelling—spoke in a famous phrase of the “trained incapacity” of a businessman to attend to anything but pecuniary profit. In fact Veblen used the phrase only once, on page...

24. What to Do

Download PDF (150.7 KB)
pp. 245-251

Our first suggestion is mild and educational. Scientists in the sizeless sciences need to start telling each other to seek substance. They need to stop believing that the translation of a problem into probability space relieves them of the need to consider oomph and loss functions. The ritualism of...

A Reader’s Guide

Download PDF (107.2 KB)
pp. 253-254

Although our approach is mostly nontechnical, we assume the reader is broadly familiar with testing, estimation, and error statistics as used in the life and human sciences. Some readers may appreciate guidance along these lines...

Notes

Download PDF (200.2 KB)
pp. 255-264

Works Cited

Download PDF (223.4 KB)
pp. 265-287

Index

Download PDF (279.2 KB)
pp. 289-320