Methods for Testing and Evaluating Survey Questionnaires (review)

Reliance on various methods to improve survey questions has grown steadily over the past two decades. Despite the increasing use of question testing and evaluation techniques among survey researchers, the theoretical and empirical literature on the topic remains relatively sparse. As the editors of Methods for Testing and Evaluating Survey Questionnaires note, existing texts universally cite the importance of conducting questionnaire pretests, but provide almost no guidance to researchers on how to do so. This monograph attempts to fill this gap in the survey methodology literature, and serves as a foundation for research on testing and evaluating survey questionnaires. The 24 papers included in this volume originated from the International Conference on Question Development, Evaluation, and Testing (QDET) held in November 2002. Following an introductory chapter by the editors, the monograph chapters cover all current areas of research in improving survey questions: cognitive interviewing, behavior coding, expert reviews, experimental designs, statistical modeling, focus groups and multi-method approaches.

The next four chapters concentrate on cognitive interviewing techniques. In Chapter 2, Willis explores the possible theoretical underpinnings of cognitive interviewing, one of the few theoretical discussions in the monograph. What makes this chapter more remarkable is that, along with the next three chapters, it points out the generally-accepted usefulness of cognitive interviewing despite a strong theoretical basis. The last of these chapters, by DeMaio and Landreth, provides a useful comparison of various cognitive interviewing techniques and the kinds of results they produce.

Chapters 6 through 8 provide examples of techniques to supplement conventional questionnaire pretesting, including analysis of behavior coding, response latency, and respondent debriefings. The chapter on van der Zouwen and Smit on behavior coding reminds researchers of the usefulness of this technique pioneered by Cannell and his colleagues at the University of Michigan more than 30 years ago. The authors show the value of analyzing behavior patterns between respondents and interviewers during interviews for diagnosing problems with questions.

The three papers comprising Part III address experimental approaches to evaluating and improving survey questions. Chapter 9 by Fowler makes the case for an experimental approach that has been around for some time, split sample designs. Through a series of experiments comparing pairs of original and alternative versions of questions administered to randomly split sample members, this research shows how such designs can indicate the magnitude of the effects of poorly designed questions on survey data. These examples also show how one evaluation technique, cognitive interviewing, is useful for identifying problems with questions, but the second technique, a split sample experiment, is needed to answer the question of how significant the problems are.

The next section of the monograph includes three papers on an emerging area of research in question testing and evaluation, statistical modeling. In Chapter 12, Biemer demonstrates how latent class analysis (LCA) can be used to identify survey questions that contribute significantly to measurement error in survey estimates. Such methods represent a relatively new approach to questionnaire testing and evaluation that hold the promise of predicting survey questions likely to produce unacceptable levels of measurement error. In this way, these techniques function as the statistical analog of qualitative techniques such as cognitive interviewing.

Parts V and VI expand the usefulness of the monograph to survey practitioners by applying testing methods to a variety of common survey contexts. Chapters 15 through 18 address various survey modes, including paper self-administered surveys, computer-assisted surveys and internet surveys. Chapters 19 through 22 provide examples of question evolution with special populations such establishment surveys, child and adolescent surveys, cross-national surveys and surveys that require questionnaire translation.

In contrast to nearly all of the preceding papers, the last three chapters illustrate multi-method approaches to questionnaire testing and evaluation. The final chapter by Forsyth, Rothgeb and Willis brings together several elements from the monograph by assessing a multiple-method, multi-stage approach to question assessment and improvement. This study...