We cannot verify your location
Browse Book and Journal Content on Project MUSE
OR

The Cult of Statistical Significance

How the Standard Error Costs Us Jobs, Justice, and Lives

Stephen T. Ziliak and Deirdre N. McCloskey

Publication Year: 2008

“McCloskey and Ziliak have been pushing this very elementary, very correct, very important argument through several articles over several years and for reasons I cannot fathom it is still resisted. If it takes a book to get it across, I hope this book will do it. It ought to.” —Thomas Schelling, Distinguished University Professor, School of Public Policy, University of Maryland, and 2005 Nobel Prize Laureate in Economics “With humor, insight, piercing logic and a nod to history, Ziliak and McCloskey show how economists—and other scientists—suffer from a mass delusion about statistical analysis. The quest for statistical significance that pervades science today is a deeply flawed substitute for thoughtful analysis. . . . Yet few participants in the scientific bureaucracy have been willing to admit what Ziliak and McCloskey make clear: the emperor has no clothes.” —Kenneth Rothman, Professor of Epidemiology, Boston University School of Health The Cult of Statistical Significance shows, field by field, how “statistical significance,” a technique that dominates many sciences, has been a huge mistake. The authors find that researchers in a broad spectrum of fields, from agronomy to zoology, employ “testing” that doesn’t test and “estimating” that doesn’t estimate. The facts will startle the outside reader: how could a group of brilliant scientists wander so far from scientific magnitudes? This study will encourage scientists who want to know how to get the statistical sciences back on track and fulfill their quantitative promise. The book shows for the first time how wide the disaster is, and how bad for science, and it traces the problem to its historical, sociological, and philosophical roots. Stephen T. Ziliak is the author or editor of many articles and two books. He currently lives in Chicago, where he is Professor of Economics at Roosevelt University. Deirdre N. McCloskey, Distinguished Professor of Economics, History, English, and Communication at the University of Illinois at Chicago, is the author of twenty books and three hundred scholarly articles. She has held Guggenheim and National Humanities Fellowships. She is best known for How to Be Human* Though an Economist (University of Michigan Press, 2000) and her most recent book, The Bourgeois Virtues: Ethics for an Age of Commerce (2006).

Published by: University of Michigan Press

Contents

pdf iconDownload PDF (144.3 KB)
pp. ix-xiv

read more

Preface

pdf iconDownload PDF (110.7 KB)
pp. xv-xviii

The implied reader of our book is a significance tester, the keeper of numerical things. We want to persuade you of one claim: that William Sealy Gosset (1876–1937)—aka “Student” of Student’s t-test—was right and that his difficult friend, Ronald A. Fisher, though a genius, was wrong...

read more

Acknowledgments

pdf iconDownload PDF (133.6 KB)
pp. xix-xxiii

We thank above all Morris Altman, who organized a session at the American Economic Association meetings in San Diego on these matters (January 2004) and then edited the articles into a special issue of the...

read more

A Significant Problem

pdf iconDownload PDF (188.7 KB)
pp. 1-22

For the past eighty years it appears that some of the sciences have made a mistake by basing decisions on statistical “significance.” Although it looks at first like a matter of minor statistical detail, it is not. Statistics, magnitudes, coefficients are essential scientific tools. No one...

read more

1. Dieting “Significance” and the Case of Vioxx

pdf iconDownload PDF (247.4 KB)
pp. 23-32

Suppose you want to help your mother lose weight and are considering two diet pills with identical prices and side effects. You are determined to choose one of the two pills for her. The first pill, named Oomph, will on average take off twenty pounds...

read more

2. The Sizeless Stare of Statistical Significance

pdf iconDownload PDF (175.5 KB)
pp. 33-41

The very word significance, emphasizing its statistical meaning, is often used prominently in an advertisement, especially for pharmaceuticals. Xanax is a product of the Upjohn Company designed to alleviate clinical anxiety. In May 1983 Upjohn ran a two-page spread...

read more

3. What the Sizeless Scientists Say in Defense

pdf iconDownload PDF (211.7 KB)
pp. 42-56

You will be uneasy. You will say again, “How can this be?” Statisticians and statistical scientists sometimes think they disagree with our strictures on null-hypothesis significance testing (Hoover and Siegler 2008; Mc- Closkey and Ziliak 2008). They are made unhappy by our radical...

read more

4. Better Practice: beta-Importance vs. alpha-“Significance”

pdf iconDownload PDF (165.2 KB)
pp. 57-61

We are not by any means the first people, even in economics, to express such indignation (Edgeworth 1885, 215; Morgenstern 1950, 92–93; Savage 1954, 1971a; Arrow 1959; Tullock 1959; De Finetti 1971; Leamer 1978; Mayer 1979; Zellner 1984)...

read more

5. A Lot Can Go Wrong in the Use of Significance Tests in Economics

pdf iconDownload PDF (191.5 KB)
pp. 62-73

We are quantitative economists. Our scientific work has concerned British industrial history and medieval agriculture, labor market statistics and the history of American charity, all viewed quantitatively. It has asked, How Much...

read more

6. A Lot Did Go Wrong in the American Economic Review during the 1980s

pdf iconDownload PDF (203.0 KB)
pp. 74-78

Table 6.1 shows the results for the 1980s, arranged from worst result to best by question. That is, • Seventy percent of the articles in the...

read more

7. Is Economic Practice Improving?

pdf iconDownload PDF (204.8 KB)
pp. 79-88

Our 1996 article reporting these findings, we repeat, had almost no impact. Horowitz and a few others have told us that they disagree. They see progress and give us credit for causing some of it. That would be nice. But lack of impact is typical in other fields subjected to such criticism, for...

read more

8. How Big Is Big in Economics?

pdf iconDownload PDF (190.2 KB)
pp. 89-97

Of course, not everyone gets everything wrong. The American Economic Review over the past two decades has been filled with superb economic science. In our opinion a good fraction of all the articles we read can be described this way—even though a supermajority make significant...

read more

9. What the Sizeless Stare Costs, Economically Speaking

pdf iconDownload PDF (170.9 KB)
pp. 98-105

You cannot “test” mechanically for nonzero along some scale that has no dimension of substance and cost. How many molecules do you suppose you share with William Shakespeare? We mean molecules in your body that were once in his? Surprisingly, the correct answer, in view...

read more

10. How Economics Stays That Way: The Textbooks and the Referees

pdf iconDownload PDF (194.5 KB)
pp. 106-122

The proximate cause of the unhappy situation in economics is that almost all the teachers of econometrics claim that statistical significance is the same thing as scientific significance. The econometrician David Hendry, for example, is famous for saying “test, test, test,” where the phrase means...

read more

11. The Not-Boring Rise of Significance in Psychology

pdf iconDownload PDF (180.3 KB)
pp. 123-130

Economists, we have noted repeatedly, are not the only scientists to fall short of real significance. Psychologists have done so for many decades now. An addiction to transforms of categorical data, a dependence on absolute criteria of Type I error, and a fetish for asterisk psychometrics have...

read more

12. Psychometrics Lacks Power

pdf iconDownload PDF (219.1 KB)
pp. 131-139

The cost of the psychological addiction to statistical significance can be measured by the “power function.” Power asks, “What in the proffered experiment is the probability of correctly rejecting the null hypothesis, concluding that the null hypothesis is indeed false when it...

read more

13. The Psychology of Psychological Significance Testing

pdf iconDownload PDF (194.7 KB)
pp. 140-153

Why have psychologists been unwilling to listen? One reason seems to be insecurity in a so-called soft or subjective field. Recall even the learned Paul Meehl, a psychological scientist as well as a philosopher, speaking of his own field as “soft.” The “hard/soft” dichotomy is surely a poor one...

read more

14. Medicine Seeks a Magic Pill

pdf iconDownload PDF (239.5 KB)
pp. 154-164

In general you wish to know the probability that your medical hypothesis, H, is true in view of the incomplete facts of the world. This is a problem of inference, inferring the likelihood of a result from data. If the symptoms of cholera start in the digestive system, then ingestion of something...

read more

15. Rothman’s Revolt

pdf iconDownload PDF (178.1 KB)
pp. 165-175

As early as 1978 the situation was sufficiently dire that two contributors to the New England Journal of Medicine, Drummond Rennie and Kenneth J. Rothman, published op-ed pieces in the journal pages about the matter (Rennie 1978; Rothman 1978). Rennie, the deputy editor of the...

read more

16. On Drugs, Disability, and Death

pdf iconDownload PDF (276.7 KB)
pp. 176-186

By 1988 the International Committee of Medical Journal Editors had been sufficiently pressured by the Rothmans and Altmans to revise their “uniform requirements for manuscripts submitted to biomedical journals.” “When possible,” the committee wrote, “quantify findings and present...

read more

17. Edgeworth’s Significance

pdf iconDownload PDF (202.8 KB)
pp. 187-192

How could such a strange scientific turn have taken place? How could it be that so many sciences became sizeless and (p < .05) hazardous to health and wealth? Francis Ysidro Edgeworth (1845–1926), the inventor of the very word...

read more

18. “Take 3sigma as Definitely Significant”: Pearson’s Rule

pdf iconDownload PDF (158.2 KB)
pp. 193-202

The argument from odds is old. Stephen Stigler notes that one might read a significance test into the clinical trial in the Book of Daniel (1:12–16), though “[i]t may be a stretch” as “the significance level and even the test statistic were left vague there...

read more

19. Who Sits on the Egg of Cuculus Canorus? Not Karl Pearson

pdf iconDownload PDF (144.0 KB)
pp. 203-206

An early case, applied to the eggs of the cuckoo bird, illustrates literally the feel of substantive as against statistical significance. “The Egg of Cuculus Canorus” (1901) by Oswald Latter was one of the first articles published in...

read more

20. Gosset: The Fable of the Bee

pdf iconDownload PDF (212.9 KB)
pp. 207-213

Karl Pearson couldn’t learn from Gosset. But Gosset couldn’t learn much from Pearson, either. “I am bound to say that I did not learn very much from his lectures,” he told Egon. “I never did from anyone’s and my mathematics were inadequate for the task,” he continued...

read more

21. Fisher: The Fable of the Wasp

pdf iconDownload PDF (251.3 KB)
pp. 214-226

If Gosset was the Bee, his difficult friend Fisher was the Wasp. Gosset patiently tried for a quarter century to teach Fisher about human relations, such as the importance of being kind and telling the truth and practicing humility and giving credit to other scientists and being accurate about...

read more

22. How the Wasp Stung the Bee and Took over Some Sciences

pdf iconDownload PDF (172.1 KB)
pp. 227-237

In 1922 Fisher wrote to Gosset, soliciting Gosset’s new table of t. Fisher was eager to put the updated table in the first edition of his Statistical Methods for Research Workers. They would call Gosset’s tables “the table of ‘Student’s’...

read more

23. Eighty Years of Trained Incapacity: How Such a Thing Could Happen

pdf iconDownload PDF (159.2 KB)
pp. 238-244

Thorstein Veblen—Oswald Veblen’s uncle, we noted, and so, astonishingly, the intellectual granduncle of Harold Hotelling—spoke in a famous phrase of the “trained incapacity” of a businessman to attend to anything but pecuniary profit. In fact Veblen used the phrase only once, on page...

read more

24. What to Do

pdf iconDownload PDF (150.7 KB)
pp. 245-251

Our first suggestion is mild and educational. Scientists in the sizeless sciences need to start telling each other to seek substance. They need to stop believing that the translation of a problem into probability space relieves them of the need to consider oomph and loss functions. The ritualism of...

read more

A Reader’s Guide

pdf iconDownload PDF (107.2 KB)
pp. 253-254

Although our approach is mostly nontechnical, we assume the reader is broadly familiar with testing, estimation, and error statistics as used in the life and human sciences. Some readers may appreciate guidance along these lines...

Notes

pdf iconDownload PDF (200.2 KB)
pp. 255-264

Works Cited

pdf iconDownload PDF (223.4 KB)
pp. 265-287

Index

pdf iconDownload PDF (279.2 KB)
pp. 289-320


E-ISBN-13: 9780472026104
E-ISBN-10: 0472026100
Print-ISBN-13: 9780472050079
Print-ISBN-10: 0472050079

Page Count: 352
Publication Year: 2008

Series Title: Economics, Cognition, and Society