The Cult of Statistical Significance (review)

Consider an item that is available from five representative outlets south of the river at 93, 92, 94, 95 and 92 cents per item. The same thing is listed for 96, 97, 95, 96 and 98 at five different outlets on the north. The average (mean) price in the north is 96.40 cents, and it is 93.20 in the south—i.e. 3.20 cents (or 3.3%) lower south of the river. The difference is considered to be very significant by a conventional statistical criterion (Student's t-test). However, if we are talking about the price of a small glass of beer, it is probably not worth crossing the river for a drink. In other words, the difference is statistically significant but of a magnitude that is of little consequence (or interest). On the other hand, if you contemplate buying $1 million worth of something and the unit price in the south is 3.3% lower on average, then you can save $33,000. So size matters, and the t-test is not a reasonable basis for either decision.

The Cult of Statistical Significance has much to say about all of this. In fact, the authors' take-home message, that the evaluation of "oomph" is often more important than "precision," is repeated over and over again until it shifts from titillating to slightly boring. The subtitle—How the Standard Error Costs Us Jobs, Justice, and Lives—indicates the scope of their mission. An early example about the manufacturer Merck and the drug Vioxx (an anti-inflammatory drug for osteoarthritis) starts with excitement on the theme, gets confusing because of the added criticism of management style and procedure (a separate problem) and unfortunately leaves the reader up in the air with respect to evaluating efficacy versus risk. The book does contain historical perspective, engaging biography and interesting anecdotes. The frontispiece is a 1908 photograph of William Sealy Gosset (1876–1937), who published under the nom de plume of Student; thus "Student's t-test" was born. Sadly, this turns out to be the only illustration.

The authors have academic careers in economics and are currently located in Chicago. A cover sheet from University of Michigan Press stated that the publication date was January 2008, but the paperback I was given last month for review warned on the front cover that it was an "advance reading copy." This was repeated on the back together with the statements, "These are uncorrected proofs of a book not yet published. Errors will be corrected and formats adjusted before the book appears in print." Many of the word errors (for example, "as" instead of "was") might be attributable to their escaping a computer spell-checker tool, but others are nonsense misspellings, sometimes even involving the names of key players in the narrative. Inexplicably, there is no index. One wonders about the wisdom of this curious production and promotion scheme for what promises to be an expensive book that may attract only a limited and special readership.