The Handbook of Research Synthesis

15. Combining Significance Levels

Cooper, H. and Hedges, L. V. (Eds.) 1994. The Handbook ofResearch Synthesis. New York: Russell Sage Foundation 15 COMBINING SIGNIFICANCE LEVELS 1. Introduction 2. Defining the p Value BETSY JANE BECKER Michigan State University CONTENTS 2.1 The p from a Single Test 2.2 The Misuse of p Values 2.3 Null Hypothesis Testing .2.4 Bayesian Perspectives 2.5 The Probability Density Function of p 3. Summarizing p Values 3.1 The Null Hypothesis for Combined Significance Tests 3.2 Alternative Hypotheses 3.2.1 The complement to Ho 3.2.2 The one-sided alternative 3.3 Misunderstandings of the Hypotheses 3.4 Advantages and Disadvantages of the Combined Significance Methods 4. Methods for Summarizing Significance Levels 4.1 Notation 4.2 Obtaining Exact p Values 4.3 Summary Methods 4.4 Four Exemplary Summaries 4.4.1 Minimum p method 4.4.2 Sum of z's method 4.4.3 Sum of logs method 4.4.4 Logit method 4.5 Methods for p Values from Discrete Statistics 216 216 216 217 217 217 218 218 218 219 219 219 220 220 221 221 221 221 222 222 222 222 224 224 215 216 STATISTICALLY DESCRIBING AND COMBINING STUDIES 5. Applications of Tests of Combined Probability 224 5.1 Teacher-Expectancy Effects 224 5.2 Validity Studies 225 5.3 An Evaluation of the Analyses 225 6. Statistical Issues Regarding Tests of Combined Significance 226 6.1 Weighting p Values 226 6.2 Power 227 7. Other Uses of p Values in Research Synthesis 227 7. I Vote-Counting 227 7.2 File-Drawer (Fail-Safe) Numbers 228 7.3 Tests and Contrasts Based on p Values 228 8. Conclusion 9. References 1. INTRODUCTION This chapter discusses methods for combining probability values (or significance levels) from independent significance tests. These "combined significance" methods have a long history (e.g., Fisher 1932; Tippett 1931) and have been studied extensively by statisticians. Several of the methods are closely related to the votecounting techniques described in the preceding chapter. The first sections of the chapter describe significance levels and briefly introduce the combined significance methods, differentiating them from parametric methods. Next, the hypotheses tested by the combined significance methods are examined and compared with those tested by the parametric methods. The methods are introduced, with a focus on four exemplars. Their application is illustrated using two data sets. Related statistical issues are briefly treated, as are other applications of probability values in research synthesis. Recommendations for the use of combined significance techniques conclude the chapter. 2. DEFINING THE p VALUE Simply stated, a significance level is a probability. Significance levels arise in the context of testing statistical hypotheses. Such hypotheses may concern relationships between variables, the effects of some treatmentes ), or more complex phenomena. On the basis of knowledge about the effects of interest the researcher formulates a research hypothesis. This hypothesis is "translated" into a statistical model or null hypothesis, 228 229 also referred to as Ho, which can be tested. The test statistics and their significance levels indicate the appropriateness of the statistical model for the population from which the data are obtained. 2.1 The p from a Single Test More precisely, a significance level is the probability of finding a test statistic (i.e., a set of sample data) as unusual or extreme as that calculated given that the null hypothesis is true. Consider a one-sided test involving an unknown parameter 8. A common formulation of a directional hypothesis about 8 would be Ho: 8:s 0 versus Ha: 8> O. Hypotheses can be tested about fixed values by defining 8 as a difference. For example, to test whether a population mean p.. is greater than 100 we can test Ho: 8= p.. -100:s0. When the directional test of significance about 8 is computed for a sample statistic to, the upper-tail onesided probability is Po= (OOf(t) dt=I-F{to)=P(t>to), Jto (15-1) where f(t) is the probability density function for the statistic t, and F(t) is the cumulative distribution function for t given that Ho is true. Significance levels are sometimes called Type I error probabilities, or simply just p values or p's. Significance levels range from zero to one, and values near zero show that the probability of obtaining the calculated test statistic value (or a more unusual value) would...