The Hand. of Res. Synthesis & Meta-Analysis, 2nd Ed.
Publication Year: 2009
Published by: Russell Sage Foundation
The past decade has been a time of enormous growth in the need for and use of research syntheses in the social and behavioral sciences. The need was given impetus by the increase in social research that began in the 1960s and continues unabated today. ...
About the Authors
PART I: Introduction
1. Research Synthesis as a Scientific Process
The moment we are introduced to science we are told it is a cooperative, cumulative enterprise. Like the artisans who construct a building from blueprints, bricks, and mortar, scientists contribute to a common edifice, called knowledge. ...
PART II: Formulating A Problem
2. Hypotheses and Problems in Research Synthesis
Texts on research methods often list sources of research ideas (see, for example, Cherulnik 2001). Sometimes ideas for research come from personal experiences, sometimes from pressing social issues. Sometimes a researcher wishes to test a theory meant to help...
3. Statistical Considerations
Research synthesis is an empirical process. As with any empirical research, statistical considerations have an influence at many points in the process. Some of these, such as how to estimate a particular effect parameter or establish its sampling uncertainty, are narrowly matters...
PART III: Searching the Literature
4. Scientific Communication and Literature Retrieval
Following analysts such as Russell Ackoff and his colleagues (1976), it is convenient to divide scientific communication into four modes: informal oral, informal written, formal oral, and formal written. The first two, exemplified by telephone conversations and mail among colleagues, are...
5. Using Reference Databases
The objective of this chapter is to introduce standard literature search tools and their use in research synthesis. Our focus is on research databases relevant to the social, behavioral, and medical sciences. We assume that the reader...
6. Grey Literature
The objective of this chapter, and of the original in the 1994 edition of this handbook, on which this one is based, is to provide methods of retrieving hard to find literature and information on particular areas of research.1 We begin by briefly outlining the many developments that have...
PART IV: Coding the Literature
7. Judging the Quality of Primary Research
Empirical studies vary in terms of the rigor with which they are conducted, and the quality of the studies comprising a research synthesis can have an impact on the validity of conclusions arising from that synthesis. However, the wide agreement on these points belies a fundamental...
8. Identifying Interesting Variables and Analysis Opportunities
Research synthesis relies on information reported in a selection of studies on a topic of interest. The purpose of this chapter is to examine the types of variables that can be coded from those studies and to outline the kinds of relationships that can be examined in the analysis of the...
9. Systematic Coding
A research synthesist has collected a set of studies that address a similar research question and wishes to code the studies to create a dataset suitable for meta-analysis. This task is analogous to interviewing, but a study rather than a person is interviewed. ...
10. Evaluating Coding Decisions
Coding is a critical part of research synthesis. It is an attempt to reduce a complex, messy, context-laden, and quantification-resistant reality to a matrix of numbers. Thus it will always remain a challenge to fit the numerical scheme to the reality, and the fit will never be perfect. ...
PART V: Statistically Describing Study Outcomes
11. Vote-Counting Procedures in Meta-Analysis
As the number of scientific studies continues to grow, it becomes increasingly important to integrate the results from these studies. One simple approach involves counting votes. In the conventional vote-counting procedure, one simply divides studies into three categories: those...
12. Effect Sizes for Continuous Data
In any meta-analysis, we start with summary data from each study and use it to compute an effect size for the study. An effect size is a number that reflects the magnitude of the relationship between two variables. For example, if a study reports the mean and standard deviation for the...
13. Effect Sizes for Dichotomous Data
In many studies measurements are made on binary (dichotomous) rather than numerical scales. Examples include studies of attitudes or opinions (the two categories for the response variable being agree or disagree with some statement), case-control studies in epidemiology (the two...
PART VI: Statistically Combining Effect Sizes
14. Combining Estimates of Effect Size
In 1896, Sir Almroth Wright—a colleague and mentor of Sir Alexander Fleming, who discovered penicillin— developed a vaccine to protect against typhoid (Susser 1977; see Roberts 1989). The typhoid vaccine was tested in several settings, and on the basis of these tests the vaccine...
15. Analyzing Effect Sizes: Fixed-Effects Models
A central question in research synthesis is whether methodological, contextual, or substantive differences in research studies are related to variation in effect-size parameters. ...
16. Analyzing Effect Sizes: Random-Effects Models
This volume considers the problem of quantitatively summarizing results from a stream of studies, each testing a common hypothesis. In the simplest case, each study yields a single estimate of the impact of some intervention. ...
17. Correcting for the Distorting Effects of Study Artifacts in Meta-Analysis
Every study has imperfections, many of which bias the results. In some cases we can define precisely what a methodologically ideal study would be like, and thus say that the effect size obtained from any real study will differ to some extent from the value that would have been...
PART VII: Special Statistical Issues and Problems
18. Effect Sizes in Nested Designs
Studies with nested designs are frequently used to evaluate the effects of social treatments, such as interventions, products, or technologies in education or public health. One common nested design assigns entire sites—often classrooms, schools, clinics, or communities—to the...
19. Stochastically Dependent Effect Sizes
Much of the literature on meta-analysis deals with analyzing effect sizes obtained from k independent studies in each of which a single treatment is compared with a control (or with a standard treatment). Because the studies are statistically independent, so are the effect sizes. ...
20. Model-Based Meta-Analysis
In this chapter I introduce model-based meta-analysis and update what is known about research syntheses aimed at examining models and questions more complex than those addressed in typical bivariate meta-analyses. I begin with the idea of model-based (or model-driven)...
PART VIII: Data Interpretation
21. Handling Messing Data
This chapter discusses what researchers can do when studies are missing the information needed for meta-analysis. Despite careful evaluation of coding decisions, researchers will find that studies in a research synthesis invariably differ in the types and quality of the information...
22. Sensitivity Analysis and Diagnostics
At every step in a research synthesis, decisions are made that can affect the conclusions and inferences drawn from that analysis. Sometimes a decision is easy to defend, such as one to omit a poor quality study from the meta-analysis based on prespecified exclusion criteria, or to...
23. Publication Bias
That the published scientific literature documents only a proportion of the results of all research carried out is a perpetual concern. Further, there is good direct and indirect evidence to suggest that the unpublished proportion may be systematically different from the published, since...
PART IX: Tying Research Synthesis to Substantive Issues
24. Advantages of Certainty and Uncertainty
A research synthesis typically is not an endpoint in the investigation of a topic. Rarely does a synthesis offer a definitive answer to the theoretical or empirical question that inspired the investigation. Instead, most research syntheses serve as way stations along a sometimes...
25. Research Synthesis and Public Policy
These three statements are based on some form of research syntheses. Each was motivated by different interests but all share a purpose—they were conducted to inform policy makers and practitioners about what works. Rather than relying on personal opinion, a consensus of experts,...
PART X: Reporting the Results
26. Visual and Narrative Interpretation
Meta-analyses in the social sciences and education often provide policy makers, practitioners, and researchers with critical new information that summarizes the central quantitative findings from a particular research literature. However, meta-analytic articles and reports are also rather...
27. Reporting Format
The other chapters in this handbook discuss the complexities and intricacies of conceptualizing and conducting research syntheses or systematic reviews. If these pieces of research are to be of most use to readers, they need to be reported in a clear way. ...
PART XI: Summary
28. Threats to the Validity of Generalized Inferences
This chapter provides a nonstatistical way of summarizing many of the main points in the preceding chapters. In particular, it takes the major assumptions outlined and translates them from formal statistical notation into ordinary English. ...
29. Potentials and Limitations
We suspect that readers of this volume have reacted to it in one of two ways: they have been overwhelmed by the number and complexity of the issues that face a research synthesist or they have been delighted to have a manual to help them through the synthesis process. ...
Appendix A: Data Sets
Appendix B: Tables
Page Count: 632
Publication Year: 2009
Edition: 2nd ed.
MUSE Marc Record: Download for The Hand. of Res. Synthesis & Meta-Analysis, 2nd Ed.