Comments

Paul N. Courant: This paper is an example of valuable work in which the results are generally negative, a genre that is vital to any scientific discipline and too often underrewarded.

Finding, as Brian A. Jacob does in earlier work cited in his paper, that the institution of accountability for test scores in the Chicago public schools led to marked improvement in performance on the math portion of the Iowa Test of Basic Skills leads him to ask a really interesting question: how was this improvement in outcomes produced? Having found a clear link between increasing the stakes associated with measurable outcomes and the attainment of those outcomes, Jacob seeks to exploit the natural experiment to uncover the production function for elementary education in an urban school district. As anyone who has looked at the literature on inputs and outputs in schools knows, this is an enormously ambitious project, and it is not surprising that even a good natural experiment conducted in a data-rich environment still leaves us uncertain about the nature of the production function(s) in schools.

Unfortunately, the only measures of technology available for this study are expenditure categories in school budgets. Thus the investigation seeks to find relationships between increases or decreases in ratios of supervisory to instructional personnel (for example) and improvements in scores on standardized tests. In Jacob's paper, these kinds of changes in input mix are interpreted as changes in the technology of production, rather than as mere changes in input levels. I am not so sure, although in any case finding reliable effects between categories of expenditure and measurable outcomes would be of great value for our understanding in general and for policy.

The discussion of equation 1 early in the paper seems to me to make too much and too little of the difference between changes in input levels and changes in technology. I think that Jacob makes too much of the difference because I would argue that all of the measures that he uses are really input measures. The data in the empirical work are expenditure data on various categories of expenditures in the schools; when we see the expenditures on supervisors increase a little and expenditures on aides decrease a little, I think we are looking at changes in inputs. Jacob wants to distinguish between the mix of expenditures and another kind of input, namely, how hard people work. He cites a paper by Robin Tepper, Susan Stone, and Melissa Roderick (hereafter TSR) that shows that students and teachers worked harder—put in more hours at school and at home in response to high-stakes testing—and also focused their work on improving scores on the high-stakes test.³⁴ This kind of input change is importantly different from expenditures on supervisors, aides, math teachers, and art teachers, but both kinds of change could be subsumed easily into a more general production function, one that depended on a set of inputs measured by quality-indexed time spent by relevant types of labor (including students and parents), as well as equipment, capital, and the usual set of things that go into production functions.

Jacob may have made too little of the distinction between inputs and technology in that he does not make any use of the formal structure of production functions to illuminate the distinction. Writing down a production function that allows for distinctions among types of inputs, and using the production function to provide at least an analytic typology, if not a full-blown theoretical model, could help us to interpret the analysis of the data. There might well be a stable technology (not that any economist seems to know it, but perhaps good school administrators are able to act as if they know it) according to which the behavior of supervisors, aides, math teachers, and music and art teachers in a given building is translated into measurable outputs. High-stakes testing would change the objective function (bigger weight on Iowa Test...