The Handbook of Research Synthesis

31. Threats to the Validity of Research Syntheses

Cooper, H. and Hedges, L. V. (Eds.) 1994. The Handbook ofResearch Synthesis. New York: Russell Sage Foundation 31 THREATS TO THE VALIDITY OF RESEARCH SYNTHESES GEORG E. MATT San Diego State University THOMAS D. COOK Northwestern University CONTENTS 1. The Promises of Research Synthesis 2. Threats to Inferences About the Existence of a Relationship Between Treatment and Outcome Classes 2.1 Unreliability in Primary Studies 2.2 The Restriction of Range in Primary Studies 2.3 Missing Effect Sizes in Primary Studies 2.4 The Unreliability of Codings in Meta-Analyses 2.5 Capitalizing on Chance in Meta-Analyses 2.6 Bias in Transforming Effect Sizes 2.7 The Lack of Statistical Independence Among Effect Sizes 2.8 The Failure to Weight Study-Level Effect Sizes Proportional to Their Precision 2.9 The Underjustified Use of Fixed or Random Effects Models 2.10 The Lack of Statistical Power 3. Threats to Inferences About the Causal Relationship Between Treatment and Outcome Classes 3.1 The Failure to Assign at Random 3.2 Deficiencies in the Implementation of Treatment Contrasts 3.3 Confounding Levels of a Moderator Variable with Substantively Irrelevant Study Characteristics 3.4 The Misspecification of Causal Mediating Relationships 4. Threats to Generalized Inferences 4.1 Unknown Sampling Probabilities Associated with the Set of Persons , Settings, Treatments, Outcomes, and Times Entering a MetaAnalysis 504 506 506 507 507 507 508 509 509 509 510 510 510 511 511 511 512 513 514 503 504 SU~y 4.2 The Underrepresentation of Prototypical Attributes 514 4.3 The Failure to Test for Heterogeneity in Effect Sizes 515 4.4 The Lack of Statistical Power for Studying Disaggregated Groups 515 4.5 The Restricted Heterogeneity of Substantively Irrelevant Characteristics 516 4.6 The Confounding of Subclasses with Substantively Irrelevant Characteristics 516 4.7 Restricted Heterogeneity in Classes 516 5. Conclusion 517 6. References 518 1. THE PROMISES OF RESEARCH SYNTHESIS Although the list of validity threats we present is relevant to research synthesis practice, it is not definitive. All threats are empirical products. Any list of threats should change as theories of method are improved and as critical discourse about research synthesis practice accumulates . Some of the threats we discuss apply to conclusions from individual studies as well as from research syntheses. Experienced research synthesists realize that they have to correct inadequacies in individual studies before combining their results across studies (Cooper 1989). Other threats apply only to research syntheses, in that they depend on how estimates from individual studies are synthesized. Yet even these threats have higher-order analogues in general principles of research design and statistics. Thus, "publication bias" can be construed as a particular instance of the more general threat of "sampling bias." However, publication bias is one concrete example of how sampling bias may be encountered in research synthesis, and we prefer to formulate our taxonomy of individual threats in forms that are as close as possible to how research synthesists actually experience them in their practice. . Research syn~esis promises to improve the quality of research conclusions in a number of different ways. One is by providing estimates of association (including causal association)·that are more precise than individual studies usually permit. Combining multiple studies increases the total sample size of observations for probing a hypothesis. But these original observations may not be the units of analysis in a meta-analysis; sometimes they will be much more reliable aggregates, like a difference between group means. Another potential benefit of research synthesis is that estimates of relationships may be less biased than those of most individual studies. The corpus of research relevant to a particular hypothesis is likely to include some studies in which the methodology is not perfect, but nonetheless outstanding. It is then possible to assign special inferential weight to studies with better methods. Moreover, subanalyses can also be made to examine how methodological quality is related to effect sizes; and having a collection of studies makes it possible to construct and defend the (difficult) argument that the magnitude of bias operating in some studies to inflate a relationship is equal to the bias operating in other studies to deflate it (Cook & Leviton 1980). While biases can also countervail within primary studies, in meta-analysis no single study needs to be unbiased. At issue is only the total bias across studies. While associations can be of many kinds, one stands out: association that is causal in the sense that deliberately manipulating one entity will lead to...