The Handbook of Research Synthesis

3. Statistical Considerations

Cooper, H. and Hedges, L. V. (Eds.) 1994. The Handbook ofResearch Synthesis. New York: Russell Sage Foundation 1. 2. 3. 4. 5. 6. 3 STATISTICAL CONSIDERATIONS Introduction Problem Formulation LARRY V. HEDGES University of Chicago CONTENTS 2.1 The Universe to Which Generalizations Are Made 2.1.1 The fixed effects (conditional) model 2.1.2 The random effects (unconditional) model 2.1.3 Fixed versus random effects 2.2 The Nature of Parameters 2.3 The Number and Source of Hypotheses Data Collection 3.1 Representativeness 3.2 Dependence Data Analysis 4.1 The Unity of Statistical Methods in Meta-Analysis 4.2 Large-Sample Approximations Conclusion References 30 30 30 30 31 32 33 34 35 35 35 36 36 36 37 37 29 30 FORMULATING A PROBLEM FOR A RESEARCH SYNTHESIS 1. INTRODUCTION Research synthesis is an empirical process. As with any empirical research, statistical considerations have an influence at many points in the process. Some of these considerations, such as how to test particular hypotheses . are narrowly matters of statistical practice. They are considered in detail in subsequent chapters of this handbook. Other issues are more conceptual and might best be considered statistical considerations that impinge on general matters of research strategy or interpretation . This chapter addresses selected issues of the latter type. 2. PROBLEM FORMULATION The formulation of the research synthesis problem has important implications for the statistical methods that may be appropriate and for the interpretation of results. Careful consideration of the questions to be addressed in the synthesis will also have implications for data collection, data evaluation, and presentation of results. In this section I discuss two broad considerations in problem formulation: the universe to which generalizations are made and the number and source of hypotheses addressed. 2.1 The Universe to Which Generalizations Are Made One of the most subtle and difficult aspects of problem formulation is the specification of the universe to which the researcher wishes to generalize. The term "universe " rather than population or hyperpopulation is used to avoid confusion with other uses of these terms in statistics.l The basic issue is one of how the results of the synthesis are to be interpreted. One perspective is the fixed effects (conditional) model, perhaps the most frequently used model of generalization in quantitative research synthesis. The other is the random effects (unconditional) model. Both perspectives have their adherents and both can be justified on logical grounds. The choice between them is a matter of choosing between two universes of possible studies that we might wish to know about. In both cases we generalize to [The use of the tenn "universe" also suggests a connection with two related problems of generalization in which this tenninology has been used: the dependability of behavioral measurements (Cronbach et al. 1972) and generalization in program evaluation (Cronbach 1982). studies "like those that have been conducted," the difference between them is in the precise definition of the term "like" and how uncertainty is treated in the inference process. To make the distinctions clear we use the terms "universe" and "sample." The universe is the hypothetical collection of studies that could be conducted in principle and about which we wish to generalize. The study sample is the ensemble of studies that are used in the review and that provide the effect size data used in the research synthesis. 2.1.1 The fixed effects (conditional) model In the fixed effects, or conditional, model, the universe to which generalizations are made consists of ensembles of studies identical to those in the study sample except for the particular people (or primary sampling units) that appear in the studies. Thus, the studies in the universe differ from those in the study sample only as a result of the sampling of people into the groups of the studies. The only source of sampling error or uncertainty is therefore the variation resulting from the sampling of people into studies. In a strict sense the universe is structured: It is a collection of ensembles of studies, each study in an ensemble corresponding to a particular study in each of the other ensembles. Each of the corresponding studies would have exactly the same effect size parameter (population effect size). In fact part of the definition of "identical" (in the requirement that corresponding studies in different ensembles of this universe be identical) is that they have the same effect size parameter. The model is called the conditional model because it can be conceived as a...