The Handbook of Research Synthesis and Meta-Analysis

16. Analyzing Effect Sizes: Random-Effects Models

295 16 ANALYZING EFFECT SIZES: RANDOM-EFFECTS MODELS STEPHEN W. RAUDENBUSH University of Chicago C O N T E N T S 16.1 Introduction 296 16.2 Rationale 297 16.2.1 The Classical View 297 16.2.2 The Bayesian View 298 16.3 The Model 298 16.3.1 Level-1 Model 298 16.3.2 Level-2 Model 299 16.3.3 Mixed Model 299 16.4 Data 299 16.5 The Logic of Inference 300 16.5.1 Do the Effect Sizes Vary? 300 16.5.2 How Large Is the Variation in True Effect Sizes? 302 16.5.2.1 A Variation to Signal Ratio 302 16.5.2.2 Plausible Value Intervals 302 16.5.2.3 Intra-Class Correlation 302 16.5.3 How Can We Make Valid Inferences About the Average Effect Size When the True Effect Sizes Vary? 303 16.5.4 Why Do Study Effects Vary? 303 16.5.5 How Effective Are Such Models in Accounting for Effect Size Variation? 304 16.5.6 What Is the Best Estimate of the Effect for Each Study? 304 16.5.6.1 Unconditional Shrinkage Estimation 304 16.5.6.1 Conditional Shrinkage Estimation 306 16.6 Summary and Conclusions 306 16.6.1 Advantages of the Approach 306 296 STATISTICALLY COMBINING EFFECT SIZES 16.1 INTRODUCTION This volume considers the problem of quantitatively summarizing results from a stream of studies, each testing a common hypothesis. In the simplest case, each study yields a single estimate of the impact of some intervention . Such an estimate will deviate from the true effect size as a function of random error because each study uses a finite sample size. What is distinctive about this chapter is that the true effect size itself is regarded as a random variable taking on different values in different studies, based on the belief that differences between the studies generate differences in the true effect sizes. This approach is useful in quantifying the heterogeneity of effects across studies, incorporating such variation into confidence intervals, testing the adequacy of models that explain this variation, and producing accurate estimates of effect size in individual studies. After discussing the conceptual rationale for the random effects model, this chapter provides a general strategy for answering a series of questions that commonly arise in research synthesis: 1. Does a stream of research produce heterogeneous results? That is, do the true effect sizes vary? 2. If so, how large is this variation? 3. How can we make valid inferences about the average effect size when the true effect sizes vary? 4. Why do study effects vary? Specifically do observable differences between studies in their target populations , measurement approaches, definitions of the treatment, or historical contexts systematically predict the effect sizes? 5. How effective are such models in accounting for effect size variation? Specifically, how much variation in the true effect sizes does each model explain? 6. Given that the effect sizes do indeed vary, what is the best estimate of the effect in each study? I illustrate how to address these questions by re-analyzing data from a series of experiments on teacher expectancy effects on pupil’s cognitive skill. My aim is to illustrate, in a comparatively simple setting, to a broad audience with a minimal background in applied statistics, the conceptual framework that guides analyses using random effects models and the practical steps typically needed to implement that framework. Although the conceptual framework guiding the analysis is straightforward, a number of technical issues must 16.6.2 Threats to Valid Statistical Inference 307 16.6.2.1 Uncertainly About the Variance 307 16.6.2.2 Failure of Parametric Assumptions 307 16.6.2.3 Problems of Model Misspecification and Capitalizing on Chance 307 16.6.2.4 Multiple Effect Sizes 308 16.6.2.5 Inferences About Particular Effect Sizes 308 16A Alternative Approaches to Point Estimation 308 16A.1 Full Maximum Likelihood 308 16A.2 Restricted Maximum Likelihood 310 16A.3 Method of Moments 311 16A.3.1 MOM Using OLS Regression at Stage 1 311 16A.3.2 MOM Using WLS with Known Weights 311 16A.4 Results 312 16B Alternative Approaches to Uncertainty Estimation 312 16B.1 Conventional Approach 312 16B.2 A Quasi-F Statistic 312 16B.3 Huber-White Variances 313 16.7 Notes 314 16.8 References 314 ANALYZING EFFECT SIZES: RANDOM-EFFECTS MODELS...