In lieu of an abstract, here is a brief excerpt of the content:

1 Dieting “Significance” and the Case of Vioxx The rationale for the 5% “accept-reject syndrome” which afflicts econometrics and other areas requires immediate attention. arnold zellner 1984, 277 The harm from the common misinterpretation of p ⫽ 0.05 as an error probability is apparent. james o. berger 2003, 4 Precision Is Nice but Oomph Is the Bomb Suppose you want to help your mother lose weight and are considering two diet pills with identical prices and side effects. You are determined to choose one of the two pills for her. The first pill, named Oomph, will on average take off twenty pounds. But it is very uncertain in its effects—at plus or minus ten pounds (you can if you wish take “plus or minus” here to signify technically “two standard errors around the mean”). Oomph gives a big effect, you see, but with a high variance. Alternatively the pill Precision will take off five pounds on average. But it is much more certain in its effects. Choosing Precision entails a probable error of plus or minus a mere one-half pound. Pill Precision is estimated, in other words, much more precisely than is Oomph, at any rate in view of the sampling schemes that measured the amount of variation in each. So which pill for Mother, whose goal is to lose weight? The problem we are describing is that the sizeless sciences—from agronomy to zoology—choose Precision over Oomph every time. 23 Being precise is not, we repeat, a bad thing. Statistical significance at some arbitrary level, the favored instrument of precision lovers, reports on a particular sort of “signal-to-noise ratio,” the ratio of the music you can hear clearly relative to the static interference. Clear signals are nice, especially so in the rare cases in which the noise of small samples and not of misspecification or other “real” errors (as Gosset put it) is your chief problem . A high signal-to-noise ratio in the matter of random samples is helpful if your biggest problem is that your sample is too small, though the clarity of the signal itself is a radically incomplete criterion for making a rational decision. The signal-to-noise ratio is calculated by dividing a measure of what one wants—the sound of a Miles Davis number, the losing of body fat, the impact of the interest rate on capital investment—by a measure of the uncertainty of the signal such as the variability caused by static interference on the radio or the random variation from a smallish sample. In diet pill terms the noise—the uncertainty of the signal, the variability—is the random effects, such as the way one person reacts to the pill by contrast with the way another person does or the way one unit of capital input interacts with the financial sector compared with some other. In formal hypothesistesting terms, the signal—the observed effect—is typically compared to a “null hypothesis,” an alternative belief. The null hypothesis is a belief used to test against the data on hand, allowing one to find a difference from it if there really is one. In the weight loss example one can choose the null hypothesis to be a literal zero effect, which is a very common choice of a null. That is, the average weight loss afforded by each diet pill is being tested against the null hypothesis, or alternative belief, that the pill in question will not take any weight at all off Mom. The formula for the signal-to-noise ratio is: Observed Effect—Hypothesized Null Effect Variation of Observed Effect Plugging in the numbers from the example yields for pill Oomph (20 ⫺0)/ 10 ⫽ 2 and for pill Precision (5 ⫺0)/0.5. ⫽ 10. In other words, the signalto -noise ratio of pill Oomph is 2 to 1 and of pill Precision 10 to 1. Precision , we find, gives a much clearer signal—five times clearer. All right, then, once more: which pill for Mother? Recall: the pills are identical in every other way, including price and side effects. “Well,” say our significance-testing, sizeless scientific colleagues, “the pill with the 24 ⱐ The Cult of Statistical Significance Dieting “Significance” and the Case of Vioxx ⱐ 25 highest signal-to-noise ratio is Precision. Precision is what scientists want and what the people, such as your mother, need. So, of course, choose Precision.” But Precision is obviously the wrong choice. Wrong for Mother’s weight management...

Share