The Handbook of Research Synthesis

23. Examining Explanatory Models Through Research Synthesis

Cooper, H. and Hedges, L. V. (Eds.) 1994. The Handbook ofResearch Synthesis. New York: Russell Sage Foundation 23 EXAMINING EXPLANATORY MODELS THROUGH RESEARCH SYNTHESIS BETSY JANE BECKER Michigan State University CHRISTINE M. SCHRAM Michigan State University CONTENTS 1. Introduction 1.1 Models 1.2 Explanation and Models 1.3 Rationale for Incorporating Models in Research Synthesis 1.4 Objectives of a Model-Driven Meta-Analysis 1.5 A History of the Use of Models in Meta-Analysis 1.5.1 Harris and Rosenthal 1.5.2 Premack and Hunter 1.5.3 Shadish 1.5.4 Becker 2. Data Evaluation 3. Data Analysis 3.1 The Form of Data and Notation 3.2 The Sampling Distribution of the Correlation Coefficient 3.3 Data for Examples 3.4 Requirements of a Data Analysis 3.5 Univariate Data Analysis 3.6 Random Effects and Mixed Model Analyses 3.6.1 Other random effects analyses 3.6.2 The EM estimation of Pj and T 3.7 Generalized Least Squares 3.7.1 A model of a common population correlation matrix 3.7.2 A test of homogeneity 3.7.3 Modeling between-studies differences in correlation matrices 3.7.4 Random effects models 3.8 Estimating a Synthesized Model 358 358 359 360 360 360 361 361 361 361 361 363 363 363 363 364 365 366 367 367 368 368 369 369 370 370 357 358 SPECIAL STATISTICAL ISSUES AND PROBLEMS 3.8.1 A fixed effects example 371 3.8.2 A random effects example 372 3.8.3 Summary 372 4. Statistical Problems in Synthesizing Models 372 4.1 Missing Data 372 4.2 Assumptions of the Analysis 372 4.2.1 The operationalization of constructs 373 4.2.2 Variation in synthetic models 373 4.2.3 Sources of artifactual variation 374 4.2.4 A lack of data and model misspecification 374 4.2.5 The issue of causality 375 4.3 A Problem-free Model-based Synthesis? 375 5. Conclusion 375 6. References 375 7. Appendix: The Posterior Distribution of p, the EM Algorithm, and Missing Data 377 1. INTRODUCTION This chapter describes a way to bring substantive theory directly into the process of research synthesis, something that critics of quantitative research synthesis have claimed is missing from it. We begin with a discussion of models and a rationale for incorporating models into research synthesis. A brief history of explanatory modeling in meta-analysis follows. Then we describe three approaches to the analysis of data in a "model-driven" synthesis. Finally, we mention some of the problems that the reviewer faces in conducting a model-driven synthesis and the factors that limit inferences based on models derived from quantitative reviews. 1.1 Models The term "model" implies a set of postulated interrelationships among constructs or variables. Models allow and encourage the simultaneous examination of multiple relationships. In many fields it is no longer sufficient to examine only a few bivariate relationships, or differences on a few outcomes. For example, studies of gender differences in science examine more complicated questions than simply whether males outperform females on standardized tests of achievement (e.g., Smith 1966). They also explore under what circumstances one group excels, and what factors predict performance. However, most meta-analyses focus on questions of main effects (see, e.g., Shadish & Sweeney, 1991). Critics of meta-analysis mention its failure to attend to possibly important moderator variables (e.g., Presby 1978) and the excessive simplicity of the questions that have been addressed (e.g., the criticism that interactions have been ignored). Thus, another reason for using models in research synthesis is in response to the growing complexity of primary research. To make sense of studies that are complex and multivariate requires an inherently multivariate approach. Such an approach is exemplified by the use of models in meta-analysis. We will use as one illustrative example a model of achievement-behavior development proposed and elaborated by Eccles and her colleagues (Eccles et al. 1983). The model postulates relationships among many variables relating to math achievement (Figure 23.1) and is a product of both theoretical and empirical work. The model, originally developed as a general model of academic choice, can also be used to organize and analyze existing research, as we will show throughout this chapter . A second example of a much simpler model is drawn from the work of Friedman (in press). Her interest is in the relationships among spatial, verbal, and mathematical abilities. A...