In lieu of an abstract, here is a brief excerpt of the content:

Cooper, H. and Hedges, L. V. (&is.) 1994. The Handbook ofResearch Synthesis. New York: Russell Sage Foundation 21 CORRECTING FOR SOURCES OF ARTIFICIAL VARIATION ACROSS STUDIES JOHN E. HUNTER Michigan State University FRANK L. SCHMIDT University ofIowa CONTENTS 1. Artifacts of Study Imperfections 1.1 Unsystematic Artifacts 1.1.1 Sampling error 1.1.2 Bad data 1.2 Systematic Artifacts 1.2.1 Single artifacts ~ .2.2 Multiple artifacts 1.2.3 A numerical illustration 2. Correcting for Attenuation 2.1 The Population Correlation: Attenuation and Disattenuation 2.2 The Sample Correlation 3. Meta-Analysis of Corrected Correlations 3.1 The Mean Correlation 3.2 Corrected Versus Uncorrected Correlations 3.3 Variance of Correlations: Procedure 4. Artifact Distributions 4.1 The Mean Correlation 4.1.1 Meta-analysis on attenuated correlations 4.1.2 Correction of the mean 4.1.3 The mean compound multiplier 4.2~ Correcting the Standard Deviation 4.2.1 The variance of the artifact multiplier 4.2.2 Decomposition of the variance 4.2.3 The "75 percent rule" 5. References 324 324 324 324 324 325 326 327 328 328 328 329 329 329 330 331 332 332 332 332 333 333 334 335 335 323 324 STATISTICALLY ANALYZING EFFECT SIZES 1. ARTIFACTS OF STUDY IMPERFECTIONS Every study has imperfections. In some cases we can define precisely what a perfect study might be, and thus we can say that the effect size value obtained from any real study will differ to some extent from the value that would have been obtained had the study been done perfectly. While it is important to control and estimate bias in isolated studies, it is even more important to reduce such errors in cumulative research reviews such as meta-analysis. Some authors have argued that meta-analysts should not correct for study imperfections because the purpose of meta-analysis is only to provide a description of study findings, not an estimate of what would have been found in a perfect study. But we argue that the errors that stem from study imperfections are artifactual in character; they stem from imperfections in our research methods, not from facts of nature. Thus, most scientific questions are better addressed by results from perfect studies than by results distorted by artifacts. For example, in correlation research the results most relevant to evaluation of a scientific theory are those that would be obtained from a study employing an infinitely large sample from the relevant population (i.e., the population itself) and employing measures of the independent and dependent variables that are free of measurement error and are perfectly construct valid. Such a study would be expected to provide a perfectly accurate estimate of the relation between constructs in the population of interest; such an estimate is maximally relevant to the testing and evaluation of scientific theories (and also to theory construction). We believe that corrections for errors in study findings due to study imperfections (which we call "artifacts") is essential to the development of cumulative knowledge. Most artifacts with which we are concerned have been studied in the field of psychometrics. If possible, we search for methods of measuring the artifact and correcting for its influence. 1.1 Unsystematic Artifacts Some artifacts produce a systematic effect on the study effect size and some cause unsystematic effects. Even within a single study, it is sometimes possible to correct for a systematic effect, though it usually requires special information to do so. Unsystematic effects usually cannot be corrected in isolated studies and mayor may not be correctable at the level of meta-analysis. The two unsystematic artifacts currently identified are sampling error and bad data. 1.1.1 Sampling error It is impossible to correct for the effect of sampling error in an isolated study. The confidence interval gives an idea of the potential size of the sampling error, but the magnitude of the sampling error in anyone study is unknown and hence cannot be corrected. The effects of sampling error can be measured and eliminated in meta-analysis if the number of studies is large enough to produce a very large total sample size. If the total sample size in the meta-analysis is not large enough, one can still correct for the effects of sampling error, though the correction is less precise and some smaller amount of sampling error will remain in the final meta-analysis results (second-order sampling error; see Hunter & Schmidt 1990b, Chapter 9). A metaanalysis that corrects for only...

Share