In lieu of an abstract, here is a brief excerpt of the content:

207 11 VOTE-COUNTING PROCEDURES IN META-ANALYSIS BRAD J. BUSHMAN MORGAN C. WANG University of Michigan and VU University University of Central Florida C O N T E N T S 11.1 Introduction 208 11.2 Counting Positive and Significant Positive Results 208 11.3 Vote-Counting Procedures 208 11.3.1 The Conventional Vote-Counting Procedure 208 11.3.2 The Sign Test 209 11.3.3 Confidence Intervals Based on Vote-Counting Procedures 210 11.3.3.1 Population Standardized Mean Difference  211 11.3.3.2 Population Correlation Coefficient  212 11.4 Combining Vote Counts and Effect Size Estimates 214 11.4.1 Population Standardized Mean Difference  214 11.4.2 Population Correlation Coefficient  215 11.5 Applications of Vote-Counting Procedures 216 11.5.1 Publication Bias 216 11.5.2 Results All in the Same Direction 217 11.5.3 Missing Data 219 11.6 Conclusions 219 11.7 Notes 220 11.8 References 220 208 STATISTICALLY DESCRIBING STUDY OUTCOMES 11.1 INTRODUCTION As the number of scientific studies continues to grow, it becomes increasingly important to integrate the results from these studies. One simple approach involves counting votes. In the conventional vote-counting procedure, one simply divides studies into three categories: those with significant positive results, those with significant negative results, and those with nonsignificant results. The category containing the most studies is declared the winner. For example, if the majority of studies examining a treatment found significant positive results, then the treatment is considered to have a positive effect. Many authors consider the conventional vote-counting procedure to be crude, flawed, and worthless (see Friedman 2001; Jewell and McCourt 2000; Lee and Bryk 1989; Mann 1994; Rafaeli-Mor and Steinberg 2002; Saroglou 2002; Warner 2001). Take, for example, the title of one article: “Why Vote-Count Reviews Don't Count” (Friedman 2001). We agree that the conventional votecounting procedure can be described in these ways. But all vote-counting procedures are not created equal. The vote-counting procedures described in this chapter are far more sophisticated than the conventional procedure. These more sophisticated procedures can have an important place in the meta-analyst’s toolbox. When authors use both vote-counting procedures and effect size procedures with the same data set, they quickly discover that vote-counting procedures are less powerful (see Dochy et al. 2003; Jewell and McCourt 2000; Saroglou 2002). However, vote-counting procedures should never be used as a substitute for effect size procedures. Research synthesists generally have access to four types of information from studies: • a reported effect size, • information that can be used to compute an effect size estimate (for example, raw data, means and standard deviations, test statistic values), • information about whether the hypothesis test found a statistically significant relationship between the independent and dependent variables, and the direction of that relationship (for example, a significant positive mean difference), and • information about only the direction of the relationship between the independent and dependent variables (for example, a positive1 mean difference). These types are rank ordered from most to least in terms of the amount of information they contain (Hedges 1986).2 Effect size procedures should be used for the studies that contain enough information to compute an effect size estimate (see chapters 12 and 13, this volume). Vote-counting procedures should be used for the studies that do not contain enough information to compute an effect size estimate but do contain information about the direction and the statistical significance of results, or that contain just the direction of results. We recommend that vote-counting procedures never be used alone unless none of the studies contain enough information to compute an effect size estimate. Rather, vote-counting procedures should be used in conjunction with effect size procedures. As we describe in section 11.4, effect size estimates and votecount estimates can be combined to obtain a more precise overall estimate of the population effect size. 11.2 COUNTING POSITIVE AND SIGNIFICANT POSITIVE RESULTS Although some studies do not contain enough information to compute an effect size estimate, they may still provide information about the magnitude of the relationship between the independent and dependent variables.3 Often this information is in the form of a report of the decision yielded by a significance test (for example, a significant positive mean difference) or in the form of the direction of the effect without regard to its statistical significance (for example, a positive mean difference). The first form of...

Share