The Workforce Investment Act: Implementation Experiences and Evaluation Findings

7 - The Challenges of Measuring Performance

177 7 the Challenges of Measuring Performance William S. Borden Mathematica Policy Research Both the WIA reauthorization process and the planning efforts of the European Social Fund (ESF) would benefit from a review of the recent experiences of performance management of employment training programs in the United States. This chapter presents an operational perspective on how performance systems are designed and implemented. It also discusses the challenges to effective performance management—challenges that are little known except to the state and federal staff managing the performance systems, and that are often not clearly understood. There is very little that is easy and straightforward about measuring program performance. Seemingly simple concepts such as enrollment, exit, employment, earnings, and whom and when to count must be defined very precisely for performance results to have meaning. This chapter assumes that the reader is familiar with WIA and its performance measures. The design and implementation of effective performance management involve many conceptual and operational issues. This analysis briefly touches on many of them to illustrate how involved the process is and to alert program managers to the areas that they need to address. Each of these issues requires more extensive discussion than the scope of this chapter allows. Performance management raises interesting and significant questions about organizational and human motivation, the dynamics of state-federal political power sharing, and the management of government programs. Policymakers tend to underestimate the challenges they face and sometimes lack the commitment necessary to make performance management processes as effective as they should be. The Employment and Training Administration (ETA) has corrected some of the problems that occurred early on, but there are still some operational 178 Borden aspects that need improvement. The pursuit of effective and fair performance management inevitably encounters challenges for which there are no easy solutions. Research on employment and training programs focuses primarily on evaluations of the impact of public investment in job training services , but there are other factors to consider when analyzing the WIA performance management system; there is a clear dichotomy between its program management objectives and its evaluative objectives. This analysis argues that some form of performance tracking and data validation is necessary for managing a complex national system of job training programs, even if the outcome data were not used to determine funding. Despite the great value of effective performance management, there are limits to using performance management data to drive funding decisions. It is also important to look beyond WIA and take a comprehensive approach to assessing performance management of job training services by examining the programs that serve special populations. Policymakers need to consider how to provide efficient and effective service to everyone, but especially people with disabilities, veterans, youth, and older workers, since the costs to serve them greatly exceed those of serving job seekers in general. This broader perspective also helps inform the debate about consolidating services under a universal program like WIA and provides the most useful information for the European Commission as it looks at performance management and service delivery alternatives. Choices must be made about whether to manage services under a more unified governance structure or as independent governance structures. In the United States, there is a somewhat confusing mix of approaches, with WIA and the Employment Service (ES) at the core and considerable fragmentation and overlap beyond that. This analysis will draw broadly on lessons learned from implementing performance measurement systems for WIA, the ES, the Senior Community Service Employment Program (SCSEP), and the Vocational Rehabilitation (VR) program at the Department of Education , among others.1 We begin the chapter with a conceptual framework for analyzing performance management issues. This includes discussion of the goals of performance systems, the limitations on measuring government pro- The Challenges of Measuring Performance 179 gram performance, and how measures are designed and defined. These concepts form the building blocks for designing a performance system. The next section of the chapter then discusses the distinction between using informal processes to manage performance and effective performance management. It covers the importance of implementing rigorous standardization, validation, and monitoring processes for effective performance management, and looks at the ETA’s great progress in this area despite continuing problems. The following section examines the challenges and benefits of involving stakeholders in the design and implementation of the performance measures. It analyzes the problems that occur when stakeholders are more concerned about meeting their goals than improving their...