Questions About Questions: Inquiries into the Cognitive Bases of Surveys

3. Direct Questioning About Comprehension in a Survey Setting

3 Direct Questioning About Comprehension in a Survey Setting ROBERT M. GROVES, NANCY H. FULTZ, and ELIZABETH MARTIN Background SUIVey questionnaires are useful measurement instruments to the extent that questions convey to the respondent the desired intent ofthe researcher. For much of sUIVey research this has been assumed to be an unproblematic part of the measurement process. Yet throughout the history of sUIVey research, there has been a stream ofmethodological research, now grown large with the application of concepts from cognitive psychology to sUIVeys, which has challenged that assumption. One of the initial cognitive acts by a sUIVey respondent upon hearing a question is the attribution of meaning to the oral presentation by the interviewer. This step ofcomprehension involves the retrieval from semantic memory of relevant facts concerning meaning of words and phrases. It also involves recognition of the implied meanings of combinations of words. These retrievals are made in the context of the sUIVey interview, in which relevant stimuli may include the topics discussed prior to this particular question, the respondent's perception of interviewer beliefs, and speech behaviors of the interviewer dUring delivery of the question. Following this lOgic, one possible source of variation in responses to a Single sUIVey question is differences in perceived intent or meaning of the question. Whereas there has been much attention paid to the impact of question wording in the sUIVey methodology literature, there have been only a few attempts to examine perceived meaning directly as a cause ofsensitivity 49 50 MEANING to question wording. One of the earliest was Cantril's (1944) study of the question: After the war is over, do you think people will have to work harder, about the same, or not so hard as before? 1. Harder 2. About the same 3. Not so hard 4. Don't know On a special subsample Cantril followed up the question with: When you said that people would have to work (harder, about the same, not so hard), were you thinking of people everywhere and in all walks of lifelaborers , white collar workers, farmers and businessmen--r did you have in mind one class or group of people in particular? Depending on the answer to this question, Cantril used other questions to attempt to understand the perceived meaning ofdifferent words or phrases in the question. (He found that one-third ofthe respondents believed "people" meant a certain class, and only a little more than one-half thought it meant everyone.) Later Nuckols (1953) asked respondents to use their own words to describe what a question was asking. He also concluded that standardization of words does not automatically imply standardization of meaning. Belson (1981) did further work along this line by asking people in reinterviews what specific phrases meant in the context ofsurvey questions. Ferber (1956) did the same thing for all respondents on an attitudinal question and found people readily expressing opinions about issues which they could not define. This fact later became a cornerstone of Converse's notion of "nonattitudes" (1970), which hypothesized that some attitudinal questions failed to measure well-defined affective states for some respondents. The result was low reliability of answers over replications of the measurement. Many of the studies above used small samples and failed to address the question of a methodology suitable for ongOing surveys. Schuman's idea of the "random probe" (1966) comes closer to such a method. He argues that a notion of the quality of answers can be obtained by follOwing a closed question, on random subsamples, with the probe, "Could you tell me a little more about that?" or "Could you explain a little about that?" This probe QUESTIONING ABOUT COMPREHENSION IN A SURVEY SETTING 51 differs from the ones mentioned above by not measuring meaning directly, but instead asking for an elaboration ofan answer. (It is not known whether this probe indeed does produce different answers than the meaning probes, but on the surface it appears to seek different information.) The final alert to problems ofshared meaning comes from the new critics of standardized survey interviewing (e.g., Mishler, 1986; Briggs, 1986; and Suchman and Jordan, chapter 12 in this volume). These researchers note that analysis of the interaction between respondents and interviewers uncovers repeated failures ofthe respondent to comprehend the question as desired by the researcher. Most of this work notes that the standardized interview (which heavily restricts both parties of the interaction to a small...