The Handbook of Research Synthesis

Foreword

FOREWORD The Handbook of Research Synthesis is an attempt to bring together in a single volume state-of-the-art descriptions of every phase of the research synthesis process, from problem formulation to report writing. Determining the content of the Handbook required us to use multiple decision criteria. First, by relying on personal experience and knowledge of previous books in the area, we identified the central decision points in conducting a research synthesis. This led to the creation of several of the major divisions of the text as well as specific chapters-for example,vote-counting procedures , combining significance levels, and correcting for sources of artificial variation across studies. Next, we examined the issues involved within the major tasks and broke them down so that covering the required information within a single chapter would be manageable. Thus, within the topic of evaluating the literature, separate chapters were given over to judging research quality ; evaluating the reliability of coding decisions, and managing meta-analytic databases, among others. We supplemented the resulting list of chapters with others on special topics that cut across the various stages of research synthesis, but were important enough to merit more focused attention. These topics included missing data, publication bias, dependent effect sizes, and using research syntheses to develop theories. Next, for each topic we identified an expert or experts and obtained their agreement to write on the topic. Finally, we held a meeting, in which chapter authors discussed their intended coverage. Ultimately, 32 chapters emerged with 43 authors drawn from at least five distinct disciplines : information science, statistics, psychology, education , and social policy analysis. Twenty-two of the authors also served as chapter reviewers along with 31 individuals who took part in the review process only. In the course of attempting to coordinate such an effort, we were faced with numerous editorial decisions. Perhaps the most critical involved how to handle overlap in coverage across chapters. Because we wanted the Handbook to be exhaustive, we defined topics for chapters very narrowly, but anticipated that authors would likely define their domains more broadly. They did. The issue for us, then, became whether to edit out the redundancy or leave it in. We chose to leave it in for several reasons. First, there is considerable insight to be gained when the same issue is treated from multiple perspectives. Second, the repetition of topics helps the reader understand some of the most pervasive problems that confront a research synthesist. Finally, the overlap allows the chapters to stand on their own, as more complete discussions of each topic, thus increasing their individual usefulness for teaching. Another important editorial issue involved mathematical notation. While some notation in meta-analysis has become relatively standard, few symbols are used universally . In addition, since each statistical chapter would contain some unique mathematical concepts, we had no guarantee that the same symbols might not be used to represent different concepts in different chapters. To minimize these problems, we proposed a notion system· and met with the authors to revise and expand it. The system is not perfect, however, and the reader may discover, when moving from one chapter to the next, that the same symbol is used to represent different xi xii FOREWORD concepts or that different symbols are used for the same concept. However, each chapter introduces its notation, and we hope that confusion has been minimized. Not only was finding a common mathematical notation a noteworthy editorial task but, given the number of diverse disciplines contributing to the Handbook, we were faced with problems involving" more general terminology . Again, we expected that occasionally authors would use different terms for the same concept and the same term for different concepts. Further, some concepts would be new to readers in any particular discipline . Therefore, we asked authors to contribute to a glossary by defining the key terms in their chapter, which we edited for style and overlap. While such an instrument is always flawed and incomplete, we hope it facilitates readers' passage through the book. Finally, a consideration that arises in any report of quantitative work concerns the conventions for deriving the numerical results given in examples. Specifically, there are several possible ways that numerical results can be rounded for presentation. For example, when final computations involve sums or quotients of intermediate results, the final numerical answer depends somewhat on whether and how the intermediate results are rounded. In the statistical examples, we have usually chosen to round intermediate computations, such as sums, to two...