The Handbook of Research Synthesis

2. Hypotheses and Problems in Research Synthesis

Cooper, H. and Hedges, L. V. (Eds.) 1994. The Handbook ofResearch Synthesis. New York: Russell Sage Foundation 2 HYPOTHESES AND PROBLEMS IN RESEARCH SYNTHESIS JUDITH A. HALL Northeastern University ROBERT ROSENTHAL Harvard University LINDA TICKLE-DEGNEN Boston University FREDERICK MOSTELLER Harvard University CONTENTS 1. Introduction 2. What We Can Learn: Comparing Primary Research with Research Synthesis 2. 1 Cause and Effect 2.2 Generalization 2.2.1 External validity 2.2.2 Units of analysis 2.3 Theory Development 3. The Directionality of Research Results 3. 1 The Use of Contrasts 3.2 Problems with Omnibus Approaches 4. Learning from Combining Studies 4.1 Combining Effect Sizes 4.1.1 Effect sizes based on counts 4.1.2 Effect sizes based on variables taking several values 4.2 Combining Significance Tests 5. Learning from Comparing Studies 5.1 Moderator Variables 5.2 Types of Moderator Variables 5.2.1 Low-inference codings 5.2.2 High-inference codings 5.2.3 Some cautions about moderators 5.3 Aggregate Analysis of Descriptive Statistics 6. Conclusion 7. References 18 18 18 19 19 20 21 21 21 22 22 22 22 23 24 24 24 25 25 25 26 26 27 27 17 18 FORMULATING A PROBLEM FOR A RESEARCH SYNTHESIS 1. INTRODUCTION In the social, behavioral, and medical sciences, investigators wish to strengthen their methods of accumulating evidence. They have been dissatisfied with the classical literature review because these reports of findings are often limited to directions of relationship among variables or to results of significance tests. More and more, reviews of the literature are moving from the traditional literary approaches to the quantitative techniques of research synthesis described in this handbook. Whenever more than one study has addressed the same conceptual hypothesis, research synthesis can be done. At the simplest level, an investigator might ask whether a study and its replication produce similar or different results, and what the net result is. At the most ambitious extreme, a research synthesis might summarize a very large literature, such as that on psychotherapy outcome studies (Smith & Glass 1977), exploring differences in the effectiveness of psychotherapy for many kinds of therapy and many kinds of outcome. A quantitative research synthesis requires only that one can identify a set of studies that meet certain criteria regarding the phenomenon in question: The synthesist must be able to define the research question to be summarized and identify which research designs will be considered appropriate and which operational definitions of key constructs (treatments, manipulations, measuring instruments) will be allowed. Two synthesists may not define the domain in the same way even though they want to study the same fundamental hypothesis. For example, one might include only randomized experiments testing the hypothesis in question, whereas another might include nonexperimental studies as well. As another example, one synthesist of clinical drug trials might include patients' self-reports of health and physical functioning, whereas another might include only studies with physiological measures. What is crucial is that the synthesist define the domain clearly and base decisions to include and exclude studies on this definition of constructs. Research synthesis can accomplish two fundamental tasks: learning from combining studies and learning from comparing studies. "Learning from combining studies" refers to finding, summarizing, and describing the already existing results of research, and "learning from comparing studies" refers to additional analyses that shed new light on variations in the phenomenon under study and on theoretical issues of causation, explanation , and construct validity. Thus, the synthesist can sometimes make inferences that go well beyond the original results. In the pages that follow, we identify the kinds of research questions that can be posed, some strengths and weaknesses of primary studies versus research synthesis , and selected problems in practice and interpretation . 2. WHAT WE CAN LEARN: COMPARING PRIMARY RESEARCH WITH RESEARCH SYNTHESIS Research synthesis extends our knowledge through the combination and comparison of primary studies. As with primary research, there are boundaries to how much we can learn from research synthesis. Appreciating these boundaries helps in formulating an appropriate hypothesis or research problem. One way to identify the boundaries of research synthesis is to relate them to the boundaries of primary research. We can define boundaries for both types of research in large part by asking the following questions: I. Cause and effect: How confident can we be that the independent variable actually affects the dependent or outcome measure? 2. Generalization: How confident can we be that the findings can be generalized beyond a small subset of...