Process-Tracing Methods: Foundations and Guidelines

Chapter 7. Turning Observations into Evidence

120 Chapter 7 Turning Observations into Evidence “There is nothing more deceptive than an obvious fact.” —A. C. Doyle 1892: 101 Empirical material needs to be evaluated before it can be admitted as evidence on which to base causal inferences. But how can we know that what we have observed is the evidence that our theory tests had predicted? How can we assess the inferential value of individual pieces of evidence? This chapter deals with the evaluation process, where raw empirical observations are assessed for their content, accuracy and probability, enabling us to use them as evidence to update our degree of confidence in the presence of the hypothesized causal mechanism. This process needs to be both transparent and open to scrutiny to ensure that it is as objective as possible. We use the term observation in this book to refer to raw data prior to the evaluation of its content and accuracy (i.e., degree of measurement error). After it has been assessed for its content and potential measurement error, we use the term evidence, indicating that it contains a certain level of inferential value.1 While numerous methodological texts discuss challenges relating to the collection and analysis of data, this chapter goes a step further by making these prescriptions compatible with the Bayesian logic of inference used in process-tracing, giving us a set of evaluative tools that can be used to assess our degree of confidence in the accuracy and content of the evidence collected in process-tracing research. The prescriptions are most applicable for theory-testing and explaining-outcome variants of process-tracing as a consequence of the development of predictions about what evidence we should find in the case, capturing the need to match predicted evidence with found evidence. Turning Observations into Evidence 121 The overall process of evaluating evidence is in many respects analogous to how evidence is admitted and evaluated within the U.S. legal system.2 In a court of law, prosecutors and defenders produce different observations that can be used to make inferences about what happened. These observations include witness accounts, technical examinations, DNA tests, and so on. In a given court case, not all of the collected observations are accepted as evidence, since they can be inaccurate and/or the evidence can be so likely that it provides little inferential leverage in making an argument plausible (V. Walker 2007). Before observations can be used in a court case, their accuracy and the probability of evidence must be evaluated. If we are dealing with a fingerprint match, we would treat a smeared print left on a surface at the scene of the crime, such as a train station, as potentially much less accurate than a print where clear ridges could be detected that would better enable comparison with a “rolled” print taken from the suspect. Regarding the probability of the evidence, a strong match between a DNA sample found on the victim and the DNA of the suspect is highly unlikely unless the two samples are the DNA of the same suspect. Combined, an accurate measure that is highly unlikely (p(e) is low) would have strong inferential weight in a court case, enabling the judge to infer with a reasonable degree of confidence that the suspect most likely was the culprit. Taken as a whole, the judge’s role is evidence evaluation—deciding which evidence is relevant and evaluating the accuracy and the probability of the evidence (V. Walker 2007: 1696). This means that a prosecutor cannot just show up in court with a gun and postulate the suspect used it to perpetrate a murder. To be admitted by the judge as evidence of a theory that a given suspect committed the crime, forensic material and/or testimony must either establish beyond a reasonable doubt or makes it highly plausible that (1) the victim was killed by a gun, (2) the gun was actually in the possession of the suspect at the time of the crime, and (3) the gun was the weapon used in the murder. In this evaluation process, the defense questions whether sources of measurement error raise doubt about the accuracy of the evidence. For example , if the testimony linking the weapon and the suspect comes from the suspect’s estranged wife, should we even admit the observation as evidence when she has a revenge motive that raises serious doubts about the veracity of her testimony? Further, what is the probability of the found evidence? For example...