Regression-Discontinuity Analysis: An Alternative to the Ex-Post Facto Experiment

Observational Studies 2 (2016) 119-128 Submitted 1960; Published reprinted, 12/16 Regression-Discontinuity Analysis: An Alternative to the Ex-Post Facto Experiment Donald L. Thistlewaite and Donald T. Campbell Editor’s Note: Donald Thistlewaite (1923-1997) was Professor of Psychology at Vanderbilt University and Donald Campbell (1916-1996) was University Professor at Lehigh University. This article was originally published in Journal of Educational Psychology, December 1960, Vol. 51, pp. 309-317. At the time the article was published, Donald Thistlewaite was at the National Merit Scholarship Corporation and Donald Campbell was at Northwestern University. The article is now in the public domain. Comments follow by leading current researchers in regression discontinuity designs: Peter Aronow, Nicole Basta, and Betz Halloran; Matias Cattaneo and Gonzalo Vazquez-Bare; Guido Imbens; Alessandra Mattei and Fabrizia Mealli; Jasjeet Sekhon and Rocı́o Titiunik; and Vivian Wong and Coady Wing. 1. Introduction While the term “ex post facto experiment” could refer to any analysis of records which provides a quasi-experimental test of a causal hypothesis, as described by Chapin (1938) and Greenwood (1945), it has come to indicate more specifically the mode of analysis in which two groups – an experimental and a control group – are selected through matching to yield a quasi-experimental comparison. In such studies the groups are presumed, as a result of matching, to have been equivalent prior to the exposure of the experimental group to some potentially change inducing event (the “experimental treatment”). If the groups differ on subsequent measures and if there are no plausible rival hypotheses which might account for the differences, it is inferred that the experimental treatment has caused the observed differences. This paper has three purposes: first, it presents an alternative mode of analysis, called regression-discontinuity analysis, which we believe can be more confidently interpreted than the ex post facto design; second, it compares the results obtained when both modes of analysis are applied to the same data; and, third, it qualifies interpretations of the ex post facto study recently reported in this journal (Thistlethwaite, 1959). Two groups of nearwinners in a national scholarship competition were matched on several background variables in the previous study in order to study the motivational effect of public recognition. The results suggested that such recognition tends to increase the favorableness of attitudes toward intellectualism, the number of students planning to seek the MD or PhD degree, the number planning to become college teachers or scientific researchers, and the number who succeed in obtaining scholarships from other scholarship granting agencies. The regressiondiscontinuity analysis to be presented here confirms the effects upon success in winning c ⃝2016 Public Domain. Thistlewaite and Campbell scholarships from other donors but negates the inference of effects upon attitudes and is equivocal regarding career plans. 2. Method 2.1 Subjects and Data1 Two groups of near-winners–5,126 students who received Certificates of Merit and 2,848 students who merely received letters of commendation–answered a questionnaire approximately 6 months after the announcement of awards in the second National Merit Scholarship program . The C of M group received greater public recognition: their names were published in a booklet distributed to colleges, universities, and other scholarship granting agencies and they received approximately two and one half times more newspaper coverage than commended students. The decision to award some students the Certificate of Merit, which meant greater public recognition, was made chiefly on the basis of “qualifying scores” on the CEEB Scholarship Qualifying Test (SQT). A second aptitude test, the Scholastic Aptitude Test, was used to confirm the high ability of all finalists, i.e., all students scoring above the SQT qualifying score for the state in which the student attended high school.2 Two hundred and forty-one students who voluntarily withdrew from the program before the second test or whose scores were not confirmed received neither award while 7,255 students who satisfactorily completed the second test received Certificates of Merit. The latter were subsequently screened by a selection committee and 827 of these students were awarded Merit Scholarships. Since the interest is in estimating the effects of honorary awards, questionnaire responses from Merit Scholars are not...