Policy Evaluation in Uncertain Economic Environments

Abstract

It will be remembered that the seventy translators of the Septuagint were shut up in seventy separate rooms with the Hebrew text and brought out with them, when they emerged, seventy identical translations. Would the same miracle be vouchsafed if seventy multiple correlators were shut up with the same statistical material? And anyhow, I suppose, if each had a different economist perched on his a priori, that would make a difference to the outcome.¹

This paper describes some approaches to macroeconomic policy evaluation in the presence of uncertainty about the structure of the economic environment under study. The perspective we discuss is designed to facilitate policy evaluation for several forms of uncertainty. For example, our approach may be used when an analyst is unsure about the appropriate economic theory that should be assumed to apply, or about the particular functional forms that translate a general theory into a form amenable to statistical analysis. As such, the methods we describe are, we believe, particularly useful in a range of macroeconomic contexts where fundamental disagreements exist as to the determinants of the problem under study. In addition, this approach recognizes that even if economists agree on the underlying economic theory that describes a phenomenon, policy evaluation often requires taking a stance on details of the economic environment, such as lag lengths and functional form, that the theory does not specify. Hence our analysis is motivated by concerns similar to those that led to the development of model calibration methods. Unlike in the usual calibration approach, however, we do not reject formal statistical inference methods but rather incorporate model uncertainty into them.

The key intuition underlying our analysis is that, for a broad range of contexts, policy evaluation can be conducted on the basis of two factors: a policymaker's preferences, and the conditional distribution of the outcomes of interest given a policy and available information. What this means is that one of the main objects of interest to scholarly researchers, namely, identification of the true or best model of the economy, is of no intrinsic importance in the policy evaluation context, even though knowledge of this model would, were it available, be very relevant in policy evaluation. Hence model selection, a major endeavor in much empirical macroeconomic research, is not a necessary component of policy evaluation.

To the contrary: our argument is that, in many cases, model selection is actually inappropriate, because conditioning policy evaluation on a particular model ignores the role of model uncertainty in the overall uncertainty that surrounds the effects of a given policy choice. This is true both in the sense that many statistical analyses of policies do not systematically evaluate the robustness of policies across different model specifications, and in the sense that many analyses fail to adequately account for the effects of model selection on statistical inference. In contrast, we advocate the use of model averaging methods, which represent a formal way through which one can avoid policy evaluation that is conditional on a particular economic model.

From a theoretical perspective, model uncertainty has important implications for the evaluation of policies. This was originally recognized in William Brainard's classic analysis,² where model uncertainty occurs in the sense that the effects of a policy on a macroeconomic outcome of interest are unknown, but may be described by the distribution of a parameter that measures the marginal effect of the policy on the outcome. Much of what we argue in terms of theory may be interpreted as a generalization of Brainard's original framework and associated insights to a broader class of model uncertainty.

An additional advantage of our approach is that it provides a firm foundation for integrating empirical analysis with policy evaluation. By explicitly casting policy evaluation exercises as the comparison of the losses associated with the distribution of macroeconomic outcomes conditional on alternative policy scenarios, connections between the observed history of the economy and policy advice are seamlessly integrated. Conventional approaches, which often equate evaluation of a policy's efficacy with the statistical significance of an estimated coefficient, do...