The Handbook of Research Synthesis

28. The Visual Presentation and Interpretation of Meta-Analyses

Cooper, H. and Hedges, L. V. (&Is.) 1994. The Handbook ofResearch Synthesis. New York: Russell Sage Foundation 28 THE VISUAL PRESENTATION AND INTERPRETATION OF META-ANALYSES RICHARD J. LIGHT JUDITH D. SINGER Harvard University Harvard University JOHN B. WILLETT Harvard University CONTENTS 1. Introduction 2. Displaying Effect Sizes: Some Examples 2.1 Current Presentations of Meta-Analytic Findings 2.2 Two Examples of the Strength of Well-Designed Displays 3. Simple Methods for Displaying Univariate Effect Sizes 3.1 Sorting Raw Data Lists by Effect Sizes 3.2 Converting Raw Data Lists into Graphic Displays 3.3, Graphic Summaries of Effect Size Distributions 4. Displaying Sources of Variation in Effect Size 4.1 Sorting Raw Data Lists by Important Study Characteristics 4.2 Converting Double-Sorted Lists into Graphic Displays 4.3 The Power of Graphs 5. Conclusion 6. References 440 440 440 441 443 443 444 444 447 448 449 450 451 453 439 440 REPORTING THE RESULTS OF RESEARCH SYNTHESIS 1. INTRODUCTION A large proportion of meta-analyses in education and the social sciences, including many that we consider excellent, pose a surprising challenge. The reader must struggle to figure out exactly what the researchers found! How are the results of meta-analyses currently presented ? Typically, and especially in the social sciences and education, readers are offered two types of summary table. The first table gives a lengthy list of the actual studies that were included in the meta-analysis, together with their date of publication, sample size, effect size, and so forth. The second table presents the overall average effect size for the entire meta-analysis. This' is often accompanied by an extended breakdown giving the mean effect sizes for important subgroups of studies that were defined by the values of various critical predictors (such as whether the original studies had a randomized or nonrandomized design, or whether the studies were published recently or not so recently). The material in these tables is not necessarily presented in any particular order. We have found it surprising that after expending great energy carrying out their research syntheses, authors of meta-analyses do not fully capitalize on these efforts. They do not display their results in ways that enhance the clarity of their conclusions. Indeed, when we examine a group of meta-analyses published in the 1990s, we ask: In this innovative age, where are the interesting displays and creative graphics? Where are the pictures and plots? It seems as though meta-analytic findings are still presented using display methods that were popular before the age of computers, graphics, and exploratory data analysis. It is helpful to readers of meta-analyses for authors to report the individual "data points," each of which is actually the outcome of a single study, in their integrative review; and that reports of research syntheses contain overall effect sizes, along with breakdowns of mean effect size along important and interesting dimensions. But is it necessary, or even advantageous, to publish these summaries as simple lists? What can be done with these lists to communicate more information? Are there more easily comprehensible and attractive ways of displaying collections of effect sizes? How can meta-analyses be reported in ways that emphasize their substantive findings? In this chapter, we give several concrete suggestions for improving the reporting and interpretation of metaanalytic findings. Our emphasis is to report findings with clarity and simplicity, and most of our suggestions can be implemented easily. In section 2 we briefly review how displays are currently used in the reporting of meta-analyses. We also illustrate how simple displays can dramatically enhance an author's presentation of findings. Since a critical problem for most meta-analyses is how to summarize all the effect sizes that were computed during the research synthesis, our section 3 focuses on constructive ways to display univariate distributions of effect size. Section 4 offers suggestions for displaying relationships between effect size and study characteristics. Included in this discussion are suggestions about how to use carefully constructed tables to present these relationships. Our theme is that simple techniques , such as ordering, or boldfacing entries in a single table, can be used to communicate critical information . And in a similar spirit, we argue that, by adopting the basic principles of exploratory data analysis, important relationships between effect size and study characteristics can be presented easily and constructively. Finally, in section 5, we suggest five guidelines for displaying results. Our broad purpose is to provide ways of ensuring that meta-analytic findings...