The Handbook of Research Synthesis

22. Stochastically Dependent Effect Sizes

Cooper, H. and Hedges, L. V. (Eds.) 1994. The Handbook ofResearch Synthesis. New York: Russell Sage Foundation 1. 2. 3. 4. 5. 6. 7. 22 STOCHASTICALLY DEPENDENT EFFECT SIZES LEON J. GLESER University of Pittsburgh INGRAM OLKIN Stanford University CONTENTS Introduction Multiple-Treatment Studies 2.1 An Example 2.2 A Regression Model 2.3 Modifications of Fonnulas Multiple-Endpoint Studies 3.1 Pooled Estimates of Variance 3.2 Nonhomogeneous Covariance Matrices Univariate Approaches 4.1 Across-Studies Inferences 4.2 Boole-Bonferroni Inequality Combining Effect Sizes Within Studies 5.1 Obtaining Composite Effect Sizes 5.2 An Example Conclusion References 340 341 342 344 345 346 349 349 349 349 351 351 352 353 354 355 339 340 SPECIAL STATISTICAL ISSUES AND PROBLEMS 1. INTRODUCTION Much of the literature on meta-analysis deals with analyzing effect sizes obtained from k independent studies in each of which a single treatment is compared with a control (or with a standard treatment). Because the studies are statistically independent, so are the effect sizes. Studies, however, are not always so simple. Thus, some studies may compare multiple variants of a treatment against a common control. For example, in a study of the beneficial effects of exercise on blood pressure, independent groups of subjects may each be assigned one of several types of exercise: running for 20 minutes daily, running for 40 minutes daily, running every other day, brisk walking, and so on. Each of these exercise groups is to be compared with a common sedentary control group. In consequence, such a study will yield more than one exercise versus control effect size. Because of the conimon control group, the estimates of these effect sizes will be correlated. Studies of this kind are called multiple-treatment studies. In other studies, the single-treatment, single-control paradigm may be followed, but multiple measures will be used as endpoints for each subject. Thus, exercise can be compared with nonexercise on systolic blood pressure, diastolic blood pressure, pulse rate, cholesterol concentration, and so on. Similarly, studies on the use of carbon dioxide for storage of apples can include measures on flavor, appearance, firmness, and resistance to disease. A treatment versus control effect size estimate may be calculated for each endpoint measure. Because measures on each subject are correlated, corresponding estimated effect sizes for these measures will be correlated within studies. Studies of this type are called multiple-endpoint studies. A special, but common, kind of multiple-endpoint study is that in which the measures (endpoints) used are subscales of a psychological test. For study-to-study comparisons, or to have a single effect size for treatment versus control, we may want to combine the effect sizes obtained from the subscales into an overall effect size. Because subscales have differing accuracies, it is well known that weighted averages of such effect sizes are required. Weighting by inverses of the variances of the estimated subscale effect sizes is appropriate when these effect sizes are independent, but may not produce the most precise estimates when the effect sizes are correlated . In each of the above situations, the existence of possible correlations among the estimated effect sizes needs to be accounted for in the analysis. To do so, additional information has to be obtained from the various studies. For example, in the multiple-endpoint studies , correlations between the endpoint measures lead to correlations between the corresponding estimated effect sizes; thus, values for these between-measures correlations will be needed for any analysis. Fortunately, as will be seen, in most cases this is all the extra information that will be needed. When the studies themselves fail to provide this information, the correlations can often be imputed from test manuals (e.g., when the measures are subscales of a test) or from published literature on the measures used. Before discussing how to deal with correlated estimated effect sizes, we need to find formulas for the correlations. Note that the dependency between estimated effect sizes in multiple-endpoint studies is intrinsic to such studies, arising from the relationships between the measures used, whereas the dependency between estimated effect sizes in multiple-treatment studies is an artifact of the design (the use of a common control). This implies that formulas for the correlations between estimated effect sizes will differ between these two types of studies, thus requiring separate consideration of each type. On the other hand, the variances of the estimated effect sizes have the same form in both types of study-namely, that obtained...