In lieu of an abstract, here is a brief excerpt of the content:

Cooper, H. and Hedges, L. V. (Eds.) 1994. The Handbook ofResearch Synthesis. New York: Russell Sage Foundation 14 YOTE-COUNTING PROCEDURES IN META-ANALYSIS BRAD J. BUSHMAN Iowa State University CONTENTS 1. Introduction 194 2. Counting Positive and Significant Positive Results 194 3. Vote-Counting Procedures 194 3.1 The Conventional Vote-Counting Procedure 194 3.2 The Sign Test 195 3.3 Confidence Intervals Based on Equal Sample Sizes 195 3.3.1 Confidence interval for the population standardized mean difference 5 197 3.3.2 Confidence interval for the population correlation coefficient p 203 3.4 Confidence Intervals Based on Unequal Sample Sizes 205 3.4.1 Confidence interval for the population standardized mean difference 5 207 3.4.2 Confidence interval for the population correlation coefficient p 208 4. Applications of Vote-Counting Methods 209 4.1 Publication Bias 209 4.2 Results All in the Same Direction 211 4.3 Missing Data 211 5. Conclusion 211 6. References 213 193 194 STATISTICALLY DESCRIBING AND COMBINING STUDIES 1. INTRODUCTION The meta-analyst generally has access to at least one of three types of data from research reports: (a) information that can be used to calculate effect size estimates (e.g., means, standard deviations, test statistic values), (b) information about whether the hypothesis tests found statistically significant relations, and (c) information about the direction of the outcomes. These data are rank ordered, from most to least, in terms of the amount of information they contain. 1 If the first type of data are available, the methods described in Chapters 18-20 of this volume are more appropriate than the methods described in this chapter. Vote-counting procedures are useful for the second and third types of data. Often the meta-analyst will want to use both effect size and votecounting procedures in the same literature review. Effect size procedures can be used for those studies that contain enough information to calculate effect size estimates. Vote-counting procedures can be used for the larger group of studies for which effect size estimates cannot be calculated. Section 2 of this chapter describes the similarities and differences between the second and third types of data outlined above. Section 3 discusses and illustrates the most common vote-counting procedures. Section 4 applies vote-counting procedures to the problems of publication bias, results all in the same direction, and missing data. Section 5 summarizes the strengths and weaknesses of vote-counting procedures. 2. COUNTING POSITIVE AND SIGNIFICANT POSITIVE RESULTS When primary research reports do not contain enough information to calculate effect size estimates, they may still provide information about the magnitude of the effect. Often the information is in the form of a report of the decision yielded by the significance test (e.g. , significant positive2 relation) or in the form of a direction of the effect without regard to its statistical significance I Hedges (1986) found that estimators based on the third type of data are, at most, only 64 percent as efficient as estimators based on the first type of data. The maximum relative efficiency occurs when the population effect size equals zero for all studies. When the population effect size does not equal zero for all studies, relative efficiency decreases as the population effect size increases and as sample size increases. 2{ use the term "positive" to refer to results in the predicted direction. (e.g., a positive mean difference or a positive correlation ). The first of these corresponds to whether the test statistic exceeds a conventional critical value at a given significance level, such as a = .05. The second corresponds to whether the test statistic exceeds the rather unconventional critical value at significance level a =.5. Both kinds of information indicate whether the test statistic exceeded a critical value Ca , the only difference is the value of a used. Thus, with vote-counting procedures , the meta-analyst counts either the number of statistically significant positive results (a = .05) or the number of positive results (a = .5). We will consider both values of a for the vote-counting procedures described in section 3. 3. VOTE-COUNTING PROCEDURES 3.1 The Conventional Vote-Counting Procedure Light and Smith (1971) were among the first to describe formally the "taking a vote" procedure: All studies which have data on a dependent variable and a specific independent variable of interest are examined. Three possible outcomes are defined. The relationship between the independent variable and the dependent variable is either significantly positive, significantly negative, or...

Share