Advances in Economic Forecasting

6 - Perspectives on Evaluating Macroeconomic Forecasts

105 6 Perspectives on Evaluating Macroeconomic Forecasts H.O. Stekler George Washington University Over the past 50 or so years, I have been concerned with the quality of economic forecasts and have written both about the procedures for evaluating these predictions and the results that were obtained from these evaluations. In this chapter I provide some perspectives on the issues involved in judging the quality of these forecasts. These include the reasons for evaluating forecasts, the questions that have been asked in these evaluations, the statistical tools that have been used, and the generally accepted results. (I do also present some new material that has not yet been published.) I do this in two parts: first focusing on shortrun gross domestic product (GDP) and inflation predictions and then turning to labor market forecasts. The process of forecasting involves a number of sequential steps. Part of that process is concerned with evaluating either the forecasts themselves or the methods that generated the predictions. This evaluationmayoccureitherwhenpastforecastsareexaminedpriortopreparing the next one or in a postmortem session to determine what went wrong and what can be learned from the errors. However, there are different perspectives or approaches for conducting these examinations. These differences may occur because some forecasts are model-based while others are derived primarily from the judgmental approach. There is a second issue. Originally, the evaluations were concerned with judging a particular model or individual. A more recent development has been to determine the value that the forecast has for the users of that prediction. The original approach calculated a variety of statistics that measured the errors of the forecasts and then compared these errors with those generated by alternative methods or individuals. The newer approach for forecast evaluation is to base it on the loss functions of the Higgins.indb 105 Higgins.indb 105 11/3/2011 10:22:56 AM 11/3/2011 10:22:56 AM 106 Stekler users (Pesaran and Skouras 2002). Elliott and Timmermann (2008), in summarizing the theoretical literature on how to evaluate forecasts, take the same approach. These studies definitely suggest that the preferred evaluation methodology utilize decision-based methods; Pesaran and Skouras, however, note that it has had limited use, and that most studies have focused on statistical measures to evaluate the skills of the forecaster or the accuracy of the model.1 There are many reasons why the decision-based methods have not been used, including technical difficulties and the huge amount of data required to describe the decision environment, particularly the loss functions of the users. The theoretical procedures provide the guidelines for undertaking forecast evaluations, but since there are problems in applying them, they have not yet yielded much information about the quality of the forecasts or an understanding of the types of errors that occur or their causes. Even though we cannot, in general, evaluate the cost of the forecast errors in the context of decision functions, I will present one particular result where it was possible to use this approach.2 Statistical measures have been the most common method for evaluating forecasts, and they have provided many insights about the quality of the forecasts and their limitations. I will, therefore, focus on that approach. These measures also provide us with the ability to obtain information about the forecasting process and why particular errors occurred . This chapter proceeds as follows. I first present a list of questions that should be addressed in any evaluation of macroeconomic forecasts; this list was taken from an old paper of mine (Stekler 1991a). That paper also presented the statistical methods that could be used to address those questions. In the intervening 20 years, forecasters have both developed new techniques for answering the original questions and asked additional questions. I will discuss these in the context of the original questions. Our macroeconomic forecast evaluations have primarily been concernedwiththepredictionsofGDPgrowthandinflation .Iwill,therefore, summarize some of the findings relating to these two variables. In making macro forecasts, economists also estimate the unemployment rate, but these forecasts have not been analyzed as extensively. Consequently, there is a limited amount of information about the quality of these forecasts. Higgins.indb 106 Higgins.indb 106 11/3/2011 10:22:56 AM 11/3/2011 10:22:56 AM Perspectives on Evaluating Macroeconomic Forecasts 107 Moreover, in analyzing labor markets in a macroeconomic growth context, long-term projections of annual employment by industry and occupation are sometimes also issued. Not...