The Handbook of Research Synthesis

20. Random Effects Models

Cooper, H. and Hedges, L. V. (Eds.) 1994. The Handbook ofResearch Synthesis. New York: Russell Sage Foundation 1. 2. 3. 4. 5. 20 RANDOM EFFECTS MODELS STEPHEN W. RAUDENBUSH Michigan State University CONTENTS Introduction Alternative Conceptual Bases of Random Effects Models for Effect Size 2.1 The Classical View 2.2 The Bayesian View Application to Parallel Randomized Experiments 3.1 A Hypothetical Example: No Between-Studies Predictors 3.1.1 Fixed effects analysis 3.1.2 Random effects analysis 3.1.3 The influence of k on the choice between fixed and random effects models 3.2 A Hypothetical Example: Incorporating Between-Studies Predictors Statistical Inference for Effect Size Data 4.1 The Model 4.2 Estimation 4.2.1 The method of moments 4.2.2 The method of maximum likelihood 4.3 Illustrative Examples 4.3.1 Example 1: Balanced data with no predictors 4.3.2 Example 2: Balanced data with one predictor 4.3.3 Example 3: Data from teacher-expectancy experiments Summary 5.1 Advantages of the Random Effects Approach 5.2 Threats to Valid Statistical Inference 302 303 303 304 305 305 305 306 307 307 308 309 310 310 311 311 311 313 314 316 316 316 301 302 STATISTICALLY ANALYZING EFFEG SIZES 5.2.1 Uncertainty about the variance 317 5.2.2 Failure of parametric assumptions 317 5.2.3 Problems of model misspecification and capitalizing on chance 317 5.2.4 Multiple effect sizes 317 5.2.5 Inferences about particular effect sizes 317 6. References 318 7. Appendix A: Estimation Procedures 318 8. Appendix B: Computing Code 320 1. INTRODUCTION A skilled researcher who wishes to assess the generalizability of findings from a single study will try to select a sample that represents the target population. The researcher will examine interaction effects-for example , by asking whether a new medical treatment depends on the age of the patient; whether the effectiveness of a new method of instruction depends on student aptitude; whether men are as responsive as women to nonverbal communication. If such interactions are null-if treatme~t effects do not depend on the age, aptitude, or sex of the subject-a finding is viewed as generalizing across these subject characteristics. If, on the other hand, such interactions are significant, the researcher can clarify the limits of generalization. Inevitably, however, the capacity of a single study to clarify the generalizability of a finding is limited. Of necessity, many conditions that might affect a finding will be constant in any given study. The manner in which the treatment is implemented, the geographic or cultural setting, the particular instrumentation used, and the historical era during which a particular expenment is implemented are all potential moderators of study findings. Such moderators can rarely be examined in the context of a single study because rarely do these factors vary within a single study. Across a series of replicated or similar studies, however , such moderators will naturally tend to vary. For example, later in this chapter we shall reanalyze data from 19 experiments testing the effect of teacher expectancy on pupil IQ. Several factors widely believed to influence results in this type of study varied across those studies: the timing and duration of the treatment, the conditions of pupil testing, and the age of the subjects. Using methods described in Chapter 19, we can test hypotheses about the effect of such potential moderators on study outcomes, thereby clarifying the conditions under which a poSItIve effect of treatment appears. However, many other characteristics of the studies varied : the year the study was conducted, its geographic location and administrative context, the characteristics of the teachers expected to implement the treatment, and the socioeconomic status of the communities providing the setting. Although no theory was available a priori to suggest that such characteristics would influence the outcome of the studies, it is at least possible that these characteristics are also related to differences between studies in effect size. For any synthesis, a long list of possible moderators of effect size can be enumerated, many of which cannot be ascertained even by the most careful reading of each study's re- -port. The multiplicity of potential moderators underlies the concept of a study's true effect size as random. Indeed, the concept of randomness in nature arises from a belief that the outcome of a process cannot be predicted in advance, precisely because the sources of influence on the outcome are both numerous and unidentifiable. The...