In lieu of an abstract, here is a brief excerpt of the content:

417 22 SENSITIVITY ANALYSIS AND DIAGNOSTICS JOEL B. GREENHOUSE SATISH IYENGAR Carnegie Mellon University University of Pittsburgh C O N T E N T S 22.1 Introduction 418 22.2 Retrieval of Literature 419 22.3 Exploratory Data Analysis 420 22.3.1 Stem-and-Leaf Plot 420 22.3.2 Box Plots 421 22.4 Combining Effect Size Estimates 423 22.4.1 Fixed Effects 423 22.4.2 Random Effects 424 22.4.3 Bayesian Hierarchical Models 426 22.4.3.1 Specification of Prior Distributions 427 22.4.4 Other Methods 428 22.5 Publication Bias 428 22.5.1 Assessing the Presence of Publication Bias 428 22.5.2 Assessing the Impact of Publication Bias 430 22.5.3 Other Approaches 431 22.6 Summary 431 22.7 Appendix 431 22.8 References 431 418 DATA INTERPRETATION 22.1 INTRODUCTION At every step in a research synthesis, decisions are made that can affect the conclusions and inferences drawn from that analysis. Sometimes a decision is easy to defend, such as one to omit a poor quality study from the metaanalysis based on prespecified exclusion criteria, or to use a weighted average instead of an unweighted one to estimate an effect size. At other times, the decision is less convincing, such as that to use only the published literature , to omit a study with an unusual effect size estimate, or to use a fixed effects instead of a random effects model. When the basis for a decision is tenuous, it is important to check whether reasonable alternatives appreciably affect the conclusions. In other words, it is important to check how sensitive the conclusions are to the method of analysis or to changes in the data. Sensitivity analysis is a systematic approach to address the question, “What happens if some aspect of the data or the analysis is changed?” When meta-analysis first came on the scene, it was greeted with considerable skepticism. Over time, with advances in methodology and with the experience due to increased use, meta-analysis has become an indispensable research tool for synthesizing evidence about questions of importance, not only in the social sciences but also in the biomedical and public health sciences (Gaver et al. 1992). Given the prominent role that meta-analysis now plays in informing decisions that have wide public health and public policy consequences, the need for sensitivity analyses to assess the robustness of the conclusions of a meta-analysis has never been greater. A sensitivity analysis need not be quantitative. For instance , the first step in a meta-analysis is to formulate the research question. Although a quantitative sensitivity analysis is not appropriate here, it is, nevertheless, useful to entertain variations of the research question before proceeding further, that is, to ask the “what if ” question. Because different questions require different types of data and data analytic methods, it is important to state clearly the aims of the meta-analysis at the outset to collect the relevant data and to explain why certain questions (and not others) were addressed. Or, at another step decisions are made concerning the identification and retrieval of studies. Here the decision is which studies to include. Should only peer-reviewed, published studies be included? What about conference proceedings or dissertations ? Should only randomized controlled studies be considered? These are decisions that a meta-analyst must make early on that can affect the generalizability of the conclusions. We view the process of asking these questions as part of sensitivity analysis because it encourages the meta-analyst to reflect on decisions and to consider the consequences of those decisions in terms of the subsequent interpretation and generalizablility of the results. Our focus in this chapter is primarily on quantitative methods for sensitivity analysis. It is important, however, to keep in mind that these early decisions about the nature of the research question and identifying and retrieving the literature can have a profound impact on the usefulness of the meta-analysis. We will use a rather broad brush to discuss sensitivity analysis. Because, as we have suggested, sensitivity analysis is potentially applicable at every step of a meta-analysis , we touch on many topics covered in greater detail elsewhere in this volume. Our aim here is to provide a unified approach to the assessment of the robustness, or cogency , of the conclusions of a research synthesis. The key element to this approach is simply the recognition of the importance of asking the “what if” question. Although...

Share