Lessons we are still learning

Observational Studies 1 (2015) 196-199 Submitted 6/15; Published 8/15 Lessons we are still learning Jennifer L. Hill jennifer.hill@nyu.edu Department of Humanities and Social Sciences New York University New York, NY 10003, USA I thoroughly enjoyed re-reading Cochran’s commentary on observational studies. In particular, Cochran captured my feelings towards the topic of observational studies quite aptly in his final sentence, “observational studies are an interesting and challenging field which demands a good deal of humility, since we can claim only to be groping toward the truth.” Scientific inquiry would benefit from greater humility among researchers pursuing causal answers today. In this comment I will briefly highlight some of the knowledge that has been gained about causal inference since that time (apologies in advance for not referencing all the scholars who have contributed – there are too many people to do it equitably). I will then focus on what I feel has been lost in the past decades and point out what I see to be important directions for the future. 1. Causal inference without randomized experiments Causal inference typically requires satisfying both structural and parametric assumptions. Randomized experiments have the advantage of addressing both of these types of assumptions . The most problematic structural assumption, ignorabilty, (also referred to as all confounders measured, selection on observables, conditional independence assumption, exchangeability , etc) is trivially satisfied in a pristine randomized experiment. Randomized experiments also ensure common support across treatment and control groups. Randomized experiments have the added advantage that they do not require conditioning on confounders for unbiased estimation, thus eliminating dependence on parametric assumptions. Moreover, even if we use a model to estimate treatment effects in this setting (for instance with goal of increasing efficiency) it is likely that our estimates will be robust to violations of the parametric assumptions of the model. Of course in practice, noncompliance, missing data, measurement issues and other complications can still wreak havoc with treatment effect estimation even in the context of a randomized experiment. Even more problematic, randomized experiments are often not possible due to ethical, financial, or logistical reasons. In the absence of a randomized experiment (or natural experiment) the structural assumptions required to identify a causal effect become more heroic, requiring appropriate conditioning on confounding covariates. Unfortunately our dependence on the parametric assumptions grows as well since we now must appropriately estimate expectations conditional on the set of proposed confounding covariates. c ⃝2015 Jennifer Hill. Lessons we are still learning Much of the work in causal inference methodology in the decades since Cochran’s paper was first published has focused on relaxation of parametric assumptions. Cochran discusses matching, subclassifcation, and covariance adjustment. Since that time, however, use of propensity scores for matching, inverse probability weighting using propensity scores have yielded improvements in our ability to estimate causal effects with less bias due to a reduced reliance on parametric assumptions. More recently, more sophisticated matching methods that capitalize on advances in computer science have increased our ability to find good balance targeted to particular balance criteria without undue investment of researcher time. Along another vein, it has been proposed that use of flexible modeling of the response surface using Bayesian nonparametrics along with appropriate checks for overlap may largely obviate the need for such preprocessing methods. All in all, our ability to condition on potential confounders without making extreme parametric assumptions has increased greatly in the past few decades. Other groundbreaking work has been done since Cochran’s paper around causal inference in longitudinal settings, greater understanding of the role of double robustness in estimation, mediation and approaches to SUTVA violations. Moreover our awareness about and ability to exploit rigourous quasi-experimental designs has grown far stronger. Most of these advances have been facilitated by the creation of a shared formal language for describing both causal estimands and assumptions. Actually, two “languages” currently hold sway: the potential outcome framework, and directed acyclic graphs, and their extensions . I will not advocate for one or another of these frameworks but rather suggest that to the extent that researchers interested in pursuing causal estimation understand these languages we can learn from each...