-
26. Bayesian Approaches to Research Synthesis
- Russell Sage Foundation
- Chapter
- Additional Information
Cooper, H. and Hedges, L. V. (Eds.) 1994. The Handbook ofResearch Synthesis. New York: Russell Sage Foundation 26 BAYESIAN APPROACHES TO RESEARCH SYNTHESIS THOMAS A. LOUIS University ofMinnesota DANIEL ZELTERMAN University ofMinnesota CONTENTS 1. Introduction 2. Basic Bayesian Methods 2.1 Two Motivating Examples 2.1.1 Example 1: Subjective opinion 2.1.2 Example 2: Objective prior evidence 2.2 The Formalism 2.3 The General (Basic) Model 3. The Compound Model 3.1 Basics 3.2 Data Analysis 4. Prior Opinion for the Compound Model 4.1 Including Study Ratings 4.2 Structuring the Evaluation 5. Data Analysis 6. Ensemble Estimates 7. Discussion 8. References 412 413 413 413 413 413 414 415 415 416 418 419 419 419 420 420 421 411 412 SPECIAL STATISTICAL ISSUES AND PROBLEMS 1. INTRODUCTION As in other scientific studies, efficient design, conduct, and analysis of a research synthesis requires that investigators use expert opinion and empirical evidence from previous studies. Design and analysis decisions are made at every stage of the process; these should be based on a combination of subjective and objective inputs. For example, statistical models are based on assumptions that require justification, but the analyst can never prove that the models are correct. The Bayesian approach provides a formal structure for incorporating such uncertainties. All unknown parameters are treated as random variables that are governed by a joint probability distribution specified prior to viewing data. These prior distributions are based on subjective opinion and objective evidence, such as the results of previous experiments. Bayesian analysis uses "Bayes's rule" to update the prior distribution in the light of the data, producing the posterior distribution. All statistical inferences (point estimates, confidence intervals, etc.) are based on this posterior distribution. Furthermore, the posterior distribution acts as the prior for the next investigation. Berger (1985), Breslow (1990), and Lindley (1990) extensively discuss Bayesian theory and practice. The Bayesian approach documents expert opinion and objective evidence on the likelihood of different values for parameters of interest such as the effect size. The approach provides a structure that can incorporate evidence and opinion on the bias, quality, and other features of each candidate study. After this specification, the Bayes approach provides automatic bias correction and weighting of the individual studies to produce a posterior distribution for the effect size of interest. This posterior distribution can be summarized by its mean, variance, interquartile range, and so on. These summaries and the entire distribution can be used to communicate the current evidence on the relative likelihood of different values of the effect size. Thus, Bayesian confidence intervals (called tolerance or prediction intervals ) are cast in a natural language: "Given the data, there is a 95 percent probability that the effect size falls in the interval (.44, .96)." This statement cannot be made unless the effect size is considered a random variable. Importantly, a meta-analysis is not conducted to inform a single individual, but to communicate the current state of information to a broad community of consumers. If the prior distributions differ substantially for different consumers, then the related Bayesian analyses can produce qualitatively as well as quantitatively different results. Therefore, it is important to perform a sensitivity analysis over the range of opinions. If conclusions are stable, then we have "findings." If they are not, the collection of Bayesian analyses underscores the finding that the data are not sufficiently compelling to bring a group of relevant consumers to consensus. This situation should motivate additional primary studies. Despite its many attractive features, the Bayesian approach has failed to gain acceptance in settings in which a large and diverse group of "consumers" constitute the target audience for the results of the experiment . Prior distributions (and therefore posterior distributions and conclusions) may vary a great deal among consumers. A broadly convincing Bayesian analysis must not depend too strongly on the prior. Granting these problems, the Bayesian approach can still be effective and convincing, especially if coupled with exploration of the impact of the prior distribution through sensitivity analysis (see Chapter 24 as well as DuMouchel & Harris 1983; Laird 1987; McCann, Hom & Kaldor 1984; Mosteller & Wallace 1964; Raudenbush & Bryk 1985) or use of other robust Bayesian methods (Berger 1985; Chaloner et al. 1993; Kadane 1986). The "what if" sensitivity analysis approach communicates clearly and should always accompany analyses based on specific prior opinions. Much has been written about the virtues of the Bayesian approach to inference, showing its superiority, in principle, over other approaches (Lindley 1990). By modeling all...