The State of the Art in Causal Inference: Some Changes Since 1972

Observational Studies 1 (2015) 182-183 Submitted 1/15; Published 8/15 The State of the Art in Causal Inference: Some Changes Since 1972 Andrew Gelman gelman@stat.columbia.edu Department of Statistics and Department of Political Science Columbia University New York, NY 10027, USA William Cochran’s 1972 article on observational studies is refreshing and includes recommendations and warnings that remain relevant today. Also interesting are the ways that Cochrans advice differs from recent textbook treatments of causal inference from observational data. Most notable, perhaps, is that Cochran talks about design, and estimation, and general goals of a study but almost nothing about causality, devoting only one page out of ten to the topic. In statistical terms, Cochran spends a lot of time on the estimator (and, more generally, the procedure to decide what estimator to use) but never defines the estimand in an observational study. He refers to bias but gives no clear sense of what exactly is being estimated (he does not, for example, define any sort of average causal effect). Modern treatments of causal inference are much more direct on this point, with the benefit of the various formal models of causal inference that were developed by Rubin and others starting in the 1970s. Scholars have pointed out the ways in which the potential-outcome formulation derives from earlier work by statisticians and economists, but Cochran’s chapter reveals what was missing in this earlier era: there was only a very weak connection between substantive concerns of designs and measurement, and statistical inference decisions regarding matching , weighting, and regression. In more recent years, the filling in of this gap has been an important research area of Rosenbaum and others; again, seeing Cochran’s essay gives us a sense of how much needed to be done. One area that Cochran discusses in detail, and which I think could use more attention in modern textbooks (including those of my collaborators and myself) is measurement. Statistics has been described as living in the intersection of variation, comparison, and measurement, and most textbooks in statistics and econometrics tend to focus on the first two of these, taking measurement for granted. Only in psychometrics do we really see measurement getting its due. So I was happy to see Cochran discuss measurement, even if he did not get to all the relevant issuesin particular, external validity, which has been the subject of much recent discussion in the context of laboratory experiments vs. field experiments vs. observational studies for social science and policy. In reading Cochran’s chapter, I was struck by his apparent lack of interest in causal identification. Modern textbooks (for example, the econometrics book of Angrist and Pischke ) discuss the search for natural experiments, along with the assumptions under which an observational study can yield valid causal inference, and various specific methods such as instrumental variables and regression discontinuity that can identify causal effects if defined c ⃝2015 Andrew Gelman. The State of the Art in Causal Inference: Some Changes Since 1972 carefully enough under specified conditions. In contrast, Cochran discusses generic beforeand -after designs and restricts himself to analysis strategies that do basic controlling for pre-treatment covariates by matching and regression. He is not so clear on what variables should be controlled for (which perhaps can be expected given that he was writing before Rubin codified the concept of ignorability), and this has the practical consequence that he devotes little space to any discussion of the data-generating process. Sure, an experiment is, all else equal, better than an observational study, but we don’t get much guidance on how an observational study can be closer or further from the experimental ideal. Cochran did write, “a claim of proof of cause and effect must carry with it an explanation of the mechanism by which the effect is produced,” which could be taken as an allusion to the substantive assumptions required for causal inference from observational data but he supplied no specifics, nothing like, for example, the exclusion restriction in instrumental variables analysis. Another topic that has appeared from time to time in the causal...