In lieu of an abstract, here is a brief excerpt of the content:

159 9 SYSTEMATIC CODING DAVID B. WILSON George Mason University C O N T E N T S 9.1 Introduction 160 9.2 Importance of Transparency and Replicability 160 9.3 Coding Eligibility Criteria 161 9.3.1 Study Features to be Explicitly Defined 161 9.3.1.1 Defining Features 161 9.3.1.2 Eligible Designs and Required Methods 162 9.3.1.3 Key Sample Features 162 9.3.1.4 Required Statistical Data 162 9.3.1.5 Geographical and Linguistic Restrictions 162 9.3.1.6 Time Frame 162 9.3.2 Refining Eligibility Criteria 163 9.4 Developing a Coding Protocol 163 9.4.1 Types of Information to Code 164 9.4.1.1 Report Identification 164 9.4.1.2 Study Setting 164 9.4.1.3 Participants 164 9.4.1.4 Methodology 165 9.4.1.5 Treatment or Experimental Manipulation(s) 165 9.4.1.6 Dependent Measures 165 9.4.1.7 Effect Sizes 166 9.4.1.8 Confidence Ratings 166 9.4.2 Iterative Refinement 166 9.4.3 Structure of the Data 168 9.4.3.1 Flat File Approach 168 9.4.3.2 Hierarchical or Relational File Approach 168 9.4.4 Coding Forms and Coding Manual 170 160 CODING THE LITERATURE 9.1 INTRODUCTION A research synthesist has collected a set of studies that address a similar research question and wishes to code the studies to create a dataset suitable for meta-analysis. This task is analogous to interviewing, but a study rather than a person is interviewed. Interrogating might be a more apt description, though it is usually the coder who loses sleep because of the study and not the other way around. The goal of this chapter is to provide synthesists with practical advice on designing and developing a coding protocol suitable to this task. A coding protocol is both the coding forms, whether paper or computerized, and the coding manual providing instructions on how to apply coding form items to studies. A well-designed coding protocol will describe the characteristics of the studies included in the research synthesis and will capture the pertinent findings in a fashion suitable for comparison and synthesis using metaanalysis . The descriptive aspect typically focuses on ways in which the studies may differ from one another. For example, the research context may vary across studies. In addition, studies may use different operationalizations of one or more of the critical constructs. Capturing variations in settings, participants, methodology, experimental manipulations, and dependent measures is an important objective of the coding protocol, not only for careful description but also for use in the analysis to explain variation in study findings (see chapters 7 and 8, this volume ). More fundamentally, the coding protocol serves as the foundation for the transparency and replicability of the synthesis. 9.2 IMPORTANCE OF TRANSPARENCY AND REPLICABILITY A bedrock of the scientific method is replication. Research synthesis exploits this feature by combining results across a collection of studies that are either pure replications of one another or conceptual replications (that is, studies examining the same basic empirical relationship , albeit with different operationalizations of key constructs and variation in method). A detailed methods section that is part of a research write-up allows others to critically assess the procedures used (transparency) and to conduct an identical or similar study to establish the veracity of the original findings (replicability). Research synthesis should in turn have these features, allowing others to duplicate the synthesis if they question the findings or wish to augment it with additional studies. This should not be interpreted to imply that research synthesis is merely the technical application of a set of procedures. The numerous decisions made in the design and conduct of a research synthesis require thoughtful scholarship. A high quality research synthesis, therefore, explicates these decisions clearly enough to allow others to fully understand the basis for the findings and to be able to replicate the synthesis. The coding protocol synthesists develop is a critical aspect of this transparency and replicability. Beyond detailing how studies will be characterized, it documents the procedures used to extract information from studies and gives guidance on how to translate the narrative information of a research report into a structured and (typically ) quantitative form. 9.5 Coding Mechanics 171 9.5.1 Paper Forms 171 9.5.2 Coding Directly into a Database 171 9.6 Disadvantages of a Spreadsheet 172 9.7 Training Coders...

Share