Disaggregating International Regimes: A New Approach to Evaluation and Comparison

2. Effectiveness Theories and Methods

Effectiveness Theories and Methods 2 At the core of regime-effectiveness studies are two questions: would problem solving be significantly lower if the regime had not existed, and how far is the problem from being fully solved? As this chapter shows, answering the first, counterfactual question requires a good explanation of why the actual level of problem solving varies. The approach taken here is to identify the main drivers of and impediments to problem solving and then examine whether those factors have been influenced by the regime. Answering the second question, on the adequacy of any regime effects, requires clear specification of what would constitute full problem solving. The first section examines how regime effectiveness can be measured in a way that is valid and determinate, and yet general enough to allow comparison across regimes and issue areas. My yardstick is an adapted version of the Oslo-Potsdam formula, which takes account of the causal effect of the regime on the problem, as well as the adequacy of that effect. Especially important here is the level of problem solving that would pertain if there were no regime. Accordingly, the subsequent section considers some objections to the use of counterfactual analysis in causal assessment and identifies general standards for doing it persuasively. Substantiating the most plausible no-regime level of problem solving requires clarifying the role of the regime within a broader set of factors that may influence problem solving. That is why I then turn to causal modeling and consider how the “mechanism approach” common in analyses of regime effectiveness can be a useful first step in identifying drivers and impediments. Among the factors that come into focus are malignancy, or tension between individual and collective interests, and collaboration, the state of knowledge, obligation, and behavioral transparency. Theoretical models must be empirically validated. In the subsequent section we will see how various techniques such as process tracing and 38 Chapter 2 comparative or statistical analysis can be applied and sometimes combined for this purpose. Among them, fuzzy set qualitative comparative analysis (QCA) is especially helpful when causality is complex and the number of observations is intermediate, as is often the case in studies of international regime effectiveness. The final substantive section explains how QCA-based analysis can help to identify causal paths—that is, certain combinations of scores on a set of causal factors—that reliably yield either high or low levels of problem solving. Whenever the regime can be shown to influence the scores on the causal factors involved, such paths to success or failure can help determine the most plausible no-regime level of problem solving, allowing us to measure how effective that regime is. Measuring Effectiveness: Causality and Adequacy What kind of yardstick is best suited for measuring regime effectiveness? Like the tools for measuring problem solving, such a yardstick should differentiate cases validly and determinately but also with high generality, because with many important questions, effectiveness has to be differentiated over time or across cases. For instance, examining the significance of certain regime properties—such as access to information, qualified majority procedures, or intrusive compliance mechanisms—for problemsolving effectiveness usually requires comparison among different regimes. The scant attention that early effectiveness studies paid to measuring effectiveness explains in part why they typically pitched comparative analysis at a high level of generality. While those studies frequently offer rich and compelling accounts of causal relationships between a regime and certain aspects of problem solving, they report few findings about the relative effectiveness of regimes or about specific conditions that will promote regime effectiveness (see Levy, Keohane, and Haas 1993; Stokke and Vidas 1996b; Young 1999b). By showing how the regimes studied make a difference to problem solving, they highlight the causality dimension of effectiveness, but they have little to say about the adequacy dimension, which concerns how far the problem still is from being fully solved. The yardstick specified below incorporates both dimensions, thereby providing a more valid, yet still general and determinate, basis for measuring regime effectiveness. Making a Difference? Measurement on the Causal Dimension During the 1990s, most attempts to measure degrees of effectiveness focused on the difference between the observable level of problem solving Effectiveness Theories and Methods 39 and some account of the no-regime situation. The instruments used for measuring that difference have ranged from qualitative expert aggregation to statistically derived coefficients. Early case study–based projects on regime effectiveness gave...