The Performance of Performance Standards

9 - Do Short-Run Performance Measures Predict Long-Run Impacts?

273 9 Do Short-Run Performance Measures Predict Long-Run Impacts? James J. Heckman Carolyn J. Heinrich Jeffrey Smith This chapter culminates the analysis in this volume by examining two closely related questions.1 The first of these is posed in the title: Do performance measures based on short-run outcomes predict long-run program impacts? If they do, then performance management systems like those in JTPA and WIA will provide incentives that enhance the economic efficiency of program operations. Put differently, if existing performance measures predict long-term impacts, then their use provides some benefits to weigh against the costs documented in earlier chapters. The second question concerns the efficiency costs of cream skimming induced by the performance standards. As noted in Chapter 3, depending on the relationship between the performance measures and net program impacts, cream skimming may be efficiency increasing (a positive relationship), efficiency decreasing (a negative relationship ), or neutral (no relationship). We address these questions in two different ways. The two analyses build on different identifying assumptions but both utilize the experimental data from the National JTPA Study (NJS) introduced in Chapter 6. The two analyses represent different ways of dealing with the fact that, absent additional assumptions, experimental data do not provide impacts for individuals, only average impacts for groups. Both strategies have important limitations, which we discuss in detail later on in the chapter. Both methods yield the same basic findings. First, the short-run labor market outcomes commonly used as performance measures do 274 Heckman, Heinrich, and Smith not predict long-run impacts. Indeed, in some cases we find a perverse relationship, indicating that the performance measures actually provide an incentive for program staff to move away from, rather than toward, economic efficiency. Second, we find little evidence of an efficiency cost associated with cream skimming; if anything, it may provide a small efficiency gain. nJS DATA We use data gathered as part of the NJS, an experimental evaluation of the JTPA program described in Chapters 2 and 4, for the analyses in this chapter. The experiment was conducted at 16 of the more than 600 JTPA training centers (which we will also refer to as sites). Table 9.1 lists the sites that volunteered to participate in the experiment and provides some descriptive statistics. Columns one through three indicate the racial/ethnic composition of the adult participant population during the study, while the fourth column indicates adult participants’ average years of schooling. The fifth and sixth columns display unemployment and poverty rates. The final three columns indicate the fraction of participants assigned to each of the three experimental treatment streams, based on the services recommended for them prior to random assignment. The classroom training in occupational skills (CT-OS) stream includes individuals who were recommended to receive CT-OS and possibly other services not including subsidized on-the-job training (OJT) at private firms. The OJT treatment stream includes individuals recommended to receive OJT and possibly other services not including CT-OS. The other services stream is a residual category that, with only a few exceptions, includes individuals not recommended to receive either CT-OS or OJT. As illustrated in Exhibit 3.17 of Orr et al. (1996), individuals in the CT-OS stream usually received classroom training whether in the form of basic education or CT-OS or both. Those in the OJT stream often did not enroll; when they did enroll they tended to receive OJT or, somewhat less often, job search assistance. Individuals in the “other” treatment stream received a wide variety of services. 275 Table 9.1 Descriptive Statistics for the 16 Sites in the national JTPA Study Fraction of participants that are: Avg. yrs. of schooling for participants Unemp. rate Poverty rate Fraction of participants assigned to: Site White Black Hispanic CT-OS stream OJT stream Other services stream Corpus Christi, TX 23.3 10.4 65.5 11.2 10.2 13.4 34.3 51.5 14.1 Cedar Rapids, IA 87.8 7.6 1.3 11.6 3.6 6.0 60.0 35.4 4.6 Coosa Valley, GA 82.1 17.1 0.6 10.7 6.5 10.7 36.1 38.1 25.7 Heartland of FL 50.2 45.7 2.8 11.4 8.5 11.3 28.9 27.1 44.0 Fort Wayne, IN 72.3 23.7 2.8 11.5 4...