# The Cult of Statistical Significance

How the Standard Error Costs Us Jobs, Justice, and Lives

Publication Year: 2008

Published by: University of Michigan Press

Series: Economics, Cognition, and Society

#### Preface

The implied reader of our book is a significance tester, the keeper of numerical things. We want to persuade you of one claim: that William Sealy Gosset (1876–1937)—aka “Student” of Student’s t-test—was right and that his difficult friend, Ronald A. Fisher, though a genius, was wrong...

#### A Significant Problem

For the past eighty years it appears that some of the sciences have made a mistake by basing decisions on statistical “significance.” Although it looks at first like a matter of minor statistical detail, it is not. Statistics, magnitudes, coefficients are essential scientific tools. No one...

#### 1. Dieting “Significance” and the Case of Vioxx

Suppose you want to help your mother lose weight and are considering two diet pills with identical prices and side effects. You are determined to choose one of the two pills for her. The first pill, named Oomph, will on average take off twenty pounds...

#### 2. The Sizeless Stare of Statistical Significance

The very word *significance*, emphasizing its statistical meaning, is often
used prominently in an advertisement, especially for pharmaceuticals.
Xanax is a product of the Upjohn Company designed to alleviate clinical
anxiety. In May 1983 Upjohn ran a two-page spread...

#### 3. What the Sizeless Scientists Say in Defense

You will be uneasy. You will say again, “How can this be?” Statisticians and statistical scientists sometimes think they disagree with our strictures on null-hypothesis significance testing (Hoover and Siegler 2008; Mc- Closkey and Ziliak 2008). They are made unhappy by our radical...

#### 4. Better Practice: beta-Importance vs. alpha-“Significance”

We are not by any means the first people, even in economics, to express such indignation (Edgeworth 1885, 215; Morgenstern 1950, 92–93; Savage 1954, 1971a; Arrow 1959; Tullock 1959; De Finetti 1971; Leamer 1978; Mayer 1979; Zellner 1984)...

#### 5. A Lot Can Go Wrong in the Use of Significance Tests in Economics

We are quantitative economists. Our scientific work has concerned British industrial history and medieval agriculture, labor market statistics and the history of American charity, all viewed quantitatively. It has asked, How Much...

#### 7. Is Economic Practice Improving?

Our 1996 article reporting these findings, we repeat, had almost no impact. Horowitz and a few others have told us that they disagree. They see progress and give us credit for causing some of it. That would be nice. But lack of impact is typical in other fields subjected to such criticism, for...

#### 8. How Big Is Big in Economics?

Of course, not everyone gets everything wrong. The *American Economic
Review* over the past two decades has been filled with superb economic
science. In our opinion a good fraction of all the articles we read can be
described this way—even though a supermajority make significant...

#### 9. What the Sizeless Stare Costs, Economically Speaking

You cannot “test” mechanically for nonzero along some scale that has no dimension of substance and cost. How many molecules do you suppose you share with William Shakespeare? We mean molecules in your body that were once in his? Surprisingly, the correct answer, in view...

#### 10. How Economics Stays That Way: The Textbooks and the Referees

The proximate cause of the unhappy situation in economics is that almost all the teachers of econometrics claim that statistical significance is the same thing as scientific significance. The econometrician David Hendry, for example, is famous for saying “test, test, test,” where the phrase means...

#### 11. The Not-Boring Rise of Significance in Psychology

Economists, we have noted repeatedly, are not the only scientists to fall short of real significance. Psychologists have done so for many decades now. An addiction to transforms of categorical data, a dependence on absolute criteria of Type I error, and a fetish for asterisk psychometrics have...

#### 12. Psychometrics Lacks Power

The cost of the psychological addiction to statistical significance can be measured by the “power function.” Power asks, “What in the proffered experiment is the probability of correctly rejecting the null hypothesis, concluding that the null hypothesis is indeed false when it...

#### 13. The Psychology of Psychological Significance Testing

Why have psychologists been unwilling to listen? One reason seems to be insecurity in a so-called soft or subjective field. Recall even the learned Paul Meehl, a psychological scientist as well as a philosopher, speaking of his own field as “soft.” The “hard/soft” dichotomy is surely a poor one...

#### 14. Medicine Seeks a Magic Pill

In general you wish to know the probability that your medical hypothesis,
*H*, is true in view of the incomplete facts of the world. This is a problem
of inference, inferring the likelihood of a result from data. If the symptoms
of cholera start in the digestive system, then ingestion of something...

#### 15. Rothman’s Revolt

As early as 1978 the situation was sufficiently dire that two contributors
to the *New England Journal of Medicine*, Drummond Rennie and Kenneth
J. Rothman, published op-ed pieces in the journal pages about the
matter (Rennie 1978; Rothman 1978). Rennie, the deputy editor of the...

#### 16. On Drugs, Disability, and Death

By 1988 the International Committee of Medical Journal Editors had been sufficiently pressured by the Rothmans and Altmans to revise their “uniform requirements for manuscripts submitted to biomedical journals.” “When possible,” the committee wrote, “quantify findings and present...

#### 18. “Take 3sigma as Definitely Significant”: Pearson’s Rule

The argument from odds is old. Stephen Stigler notes that one might read a significance test into the clinical trial in the Book of Daniel (1:12–16), though “[i]t may be a stretch” as “the significance level and even the test statistic were left vague there...

#### 19. Who Sits on the Egg of *Cuculus Canorus*? Not Karl Pearson

An early case, applied to the eggs of the cuckoo bird, illustrates literally
the feel of substantive as against statistical significance. “The Egg of
*Cuculus Canorus*” (1901) by Oswald Latter was one of the first articles
published in...

#### 20. Gosset: The Fable of the Bee

Karl Pearson couldn’t learn from Gosset. But Gosset couldn’t learn much from Pearson, either. “I am bound to say that I did not learn very much from his lectures,” he told Egon. “I never did from anyone’s and my mathematics were inadequate for the task,” he continued...

#### 21. Fisher: The Fable of the Wasp

If Gosset was the Bee, his difficult friend Fisher was the Wasp. Gosset patiently tried for a quarter century to teach Fisher about human relations, such as the importance of being kind and telling the truth and practicing humility and giving credit to other scientists and being accurate about...

#### 22. How the Wasp Stung the Bee and Took over Some Sciences

In 1922 Fisher wrote to Gosset, soliciting Gosset’s new table of *t*. Fisher
was eager to put the updated table in the first edition of his *Statistical
Methods for Research Workers*. They would call Gosset’s tables “the table
of ‘Student’s’...

#### 23. Eighty Years of Trained Incapacity: How Such a Thing Could Happen

Thorstein Veblen—Oswald Veblen’s uncle, we noted, and so, astonishingly, the intellectual granduncle of Harold Hotelling—spoke in a famous phrase of the “trained incapacity” of a businessman to attend to anything but pecuniary profit. In fact Veblen used the phrase only once, on page...

#### 24. What to Do

Our first suggestion is mild and educational. Scientists in the sizeless sciences need to start telling each other to seek substance. They need to stop believing that the translation of a problem into probability space relieves them of the need to consider oomph and loss functions. The ritualism of...

E-ISBN-13: 9780472026104

E-ISBN-10: 0472026100

Print-ISBN-13: 9780472050079

Print-ISBN-10: 0472050079

Page Count: 352

Publication Year: 2008

Series Title: Economics, Cognition, and Society

*See more Books in this Series*

MUSE Marc Record: Download for The Cult of Statistical Significance