The Handbook of Research Synthesis and Meta-Analysis

28. Threats to the Validity of Generalized Inferences

537 28 THREATS TO THE VALIDITY OF GENERALIZED INFERENCES GEORG E. MATT THOMAS D. COOK San Diego University Northwestern University C O N T E N T S 28.1 Introduction 538 28.1.1 Why We Conduct Research Syntheses 539 28.1.2 Validity Threats 540 28.1.3 Generalized Inferences 540 28.1.3.1 Demonstrating Proximal Similarity 541 28.1.3.2 Exploring Heterogeneous and Substantively Irrelevant Third Variables 541 28.1.3.3 Probing Discriminant Validity 541 28.1.3.4 Studying Empirical Interpolation and Extrapolation 541 28.1.3.5 Building on Causal Explanations 542 28.2 Threats to Inferences About the Existence of an Association Between Treatment and Outcome Classes 543 28.2.1 Unreliability in Primary Studies 543 28.2.2 Restriction of Range in Primary Studies 543 28.2.3 Missing Effect Sizes in Primary Studies 544 28.2.4 Unreliability of Codings in Meta-Analyses 544 28.2.5 Capitalizing on Chance in Meta-Analyses 545 28.2.6 Biased Effect-Size Sampling 545 28.2.7 Publication Bias 545 28.2.8 Bias in Computing Effect Sizes 546 28.2.9 Lack of Statistical Independence 546 28.2.10 Failure to Weight Effect Sizes Proportionally to Their Precision 547 28.2.11 Underjustified Use of Fixed or Random Effects Models 547 28.2.12 Lack of Statistical Power for Detecting an Association 548 538 SUMMARY 28.1 INTRODUCTION This chapter provides a nonstatistical way of summarizing many of the main points in the preceding chapters. In particular, it takes the major assumptions outlined and translates them from formal statistical notation into ordinary English. The emphasis is on expressing specific violations of formal meta-analytic assumptions as concretely labeled threats to valid inference. This explicitly integrates statistical approaches to meta-analysis with a falsificationist framework that stresses how secure knowledge depends on ruling out alternative interpretations. Thus, we aim to refocus readers’ attention on the major rationales for research synthesis and the kinds of knowledge meta-analysts seek to achieve. The special promise of meta-analysis is to foster empirical knowledge about general associations, especially causal ones, that is more secure than what other methods typically warrant. In our view, no rationale for meta-analysis is more important than its ability to identify the realm of application of a knowledge claim—that is, identifying whether the association holds with specific populations of persons, settings, times and ways of varying the cause or measuring the effect; holds across different populations of people, settings, times, and ways of operationalizing a cause and effect; and can even be extrapolated to other populations of people, settings, times, causes, and effects than those studied to date. These are all generalization tasks that researchers face, perhaps no one more explicitly than the meta-analyst. It is easy to justify why we translate violated statistical assumptions into threats to validity, particularly threats to the validity of conclusions regarding the generality of an association. The past twenty-five years of meta-analytic practice have amply demonstrated that primary studies rarely present a census or even a random sample of the populations, universes, categories, classes, or entities (terms we use interchangeably) about which generalizations are sought. The salient exception is when random sampling occurs from some clearly designated universe, a procedure that does warrant valid generalization to the population from which the sample was drawn, usually a 28.3 Threats to Inferences About the Causal Nature of an Association Between Treatment and Outcome Classes 548 28.3.1 Absence of Studies with Successful Random Assignment 549 28.3.2 Primary Study Attrition 549 28.4 Threats to Generalized Inferences 549 28.4.1 Sampling Biases Associated with Persons, Treatments, Outcomes, Settings, and Times 549 28.4.2 Underrepresentation of Prototypical Attributes 550 28.4.3 Restricted Heterogeneity of Substantively Irrelevant Third Variables 550 28.4.4 Mono-Operation Bias 551 28.4.5 Mono-Method Bias 551 28.4.6 Rater Drift 551 28.4.7 Reactivity Effects 551 28.4.8 Restricted Heterogeneity in Universes of Persons, Treatments, Outcomes, Settings, and Times 552 28.4.9 Moderator Variable Confounding 552 28.4.10 Failure to Test for Homogeneity of Effect Sizes 553 28.4.11 Lack of Statistical Power for Homogeneity Tests 554 28.4.12 Lack of Statistical Power for Studying Disaggregated Groups 554 28.4.13 Misspecification of Causal Mediating Relationships 554 28.4.14 Misspecification Models for Extrapolation 555 28.5 Conclusions 555 28.6 References...