Improving Forecasts of State Failure

I. Introduction

STATE failure" refers to the complete or partial collapse of state authority, such as occurred in Somalia and Bosnia. Failed states have governments with little political authority or ability to impose the rule of law. They are usually associated with widespread crime, violent conflict, or severe humanitarian crises, and they may threaten the stability of neighboring countries. States that sponsor international terrorism or allow it to be organized from within their borders are all failed states. Since the consequences for the citizens of these states can be very severe and the costs to the international community of rebuilding the states are often substantial, there has long been considerable interest in developing methods of risk assessment and early warning systems in the hope that foreign aid could be directed to prevent states from failing. In 1994, with these goals in mind, the U.S. government, at the behest of Vice President Gore, established and funded the State Failure Task Force, a panel of distinguished academic social scientists, experts in data collection, and consultants in statistical methods. Although the task force does not use classified information, the data amassed are nonetheless impressive: more than a thousand variables, each carefully collected and documented and many with value added beyond what is available from other sources. (See Appendix 1.) The task force, still in operation, has produced over two hundred pages of widely distributed formal reports and analyses ¹ and several published article-length summaries. ² This work has received attention in the popular news media ³ and "has gained substantial visibility and credibility among those responsible for the analysis of global security and for planning U.S. foreign policy," ⁴ an uncommon achievement for quantitative analyses in this field.

The task force reports were aimed at policymakers, but the research has been of considerable interest to the scholarly community as well. The authors make stunning claims about their success at forecasting these highly heterogeneous and idiosyncratic events, and they draw numerous important inferences about the causes of a critical and understudied political phenomenon. In this article we provide the first independent scholarly evaluation of the methods, analyses, and claims of the State Failure Task Force. We first identify and correct several methodological errors and then show how to use the task force's data to improve forecasts of state failure substantially beyond even appropriately corrected versions of its statistical models. We hope that this article can then help to connect the goals and efforts of the policy and academic communities in understanding and perhaps even addressing this critical global problem. The work analyzed here also touches on an unusually wide range of underutilized methods and relevant methodological issues; we seek to clarify some of these so that scholars can use them more productively.

II. Task Force Data and Models

According to the task force, a "state failure" consists of revolutionary wars ("sustained military conflicts between insurgents and central governments, aimed at displacing the regime"), genocides and politicides ("sustained policies by states or their agents and, in civil wars, by contending authorities that result in the deaths of a substantial portion of members of communal or political groups"), and adverse or disruptive regime transitions ("major, abrupt shifts in patterns of governance, including state collapse, periods of severe regime instability, and shifts toward authoritarian rule)." ⁵ The authors intentionally included fairly diverse events in their definition of state failure in order to follow the guidelines of policymakers as articulated to the task force. This may be a reasonable starting point, in part, because it increases the number of events in the data set, but also because it assumes that the benefits of having more events outweigh the costs of lower predictive ability and model incoherence resulting from increased heterogeneity in the outcome variable. In this article we use the dependent variable as conceptualized and measured by the task force (in order to isolate the effects of our methodological...