-
9. Asking a New Question, Getting New Answers: Evaluating Results
- Liberty Fund
- Chapter
- Additional Information
[ 152 ] 9 Asking a New Question, Getting New Answers: Evaluating Results The three large assertions of this book so far have been that we ought to use the pursuit of happiness as the criterion of success in making social policy, that the design of policy solutions must reflect one’s understanding of human nature, and that these things constitute not just a theoretical exercise but something that policy analysts ought actually to do as they go about their work. It is time to take up the last assertion, that all this has some relevance to the real task of devising better policies. In this and the following chapters I will discuss two specific ways in which the framework I have presented might be employed in assessing policy. This chapter will deal with the problem of evaluation: the art of measuring whether we are making progress, of deciding what (and how much) has been accomplished by a policy. The next chapter will begin to take up the design question: Given the existence of a social problem, how is one to divine a solution? Because the way we design solutions is so dependent on the way we assess results, I begin with the evaluation function. The Inevitability of Evaluating Despite the frayed reputation of social scientists as interpreters of social policy and its effects, there’s no way around it: People are going to make claims about the effects of social programs, and they are going to be based, for better or worse, on specific measures, specifically operationalized, of what has been accomplished. The fundamental question that has always been asked by presidents and congresses alike asking a new question: evaluating r esults [ 153 ] is “Will policy X produce the intended result Y ?” The more precisely one attempts to answer it, the more one is driven toward methods very like the ones that social scientists use. Policy analysis (as I will refer generically to this type of social science ) is predominantly quantitative. The techniques are numerous, each technique has its own idiosyncrasies, and the debates about technical issues are unending. But the esoterica of statistical analysis are not our concern. We will focus on a basic issue that receives too little attention, what I will call “the dependent variable problem.” the dependent variable problem Seen through the lens of evaluation statistics, the world is divided into “dependent” variables and “independent” variables. If one thinks in terms of cause-effect relationships, the independent variable is the cause, the dependent variable the effect. (A mnemonic for keeping them straight: An effect “depends upon” its cause, hence is dependent .) If one thinks in terms of social policy goals to be attained by social programs, the program is the independent variable (or “the intervention ”) and the thing-to-be-attained is the dependent variable. For those who think in terms of conventional statistical notation, independent variables are the Xs and dependent variables are the Ys. In a graph, the independent variable is shown on the horizontal axis, the dependent variable on the vertical axis. The social scientist’s assessment of whether progress is being made depends on the measuring stick he employs for the dependent variable. Almost all of the technical debates about the results of evaluation focus on the difficulties associated with the actual data gathered and analyzed. (Is the measure valid and reliable in the statistical sense of those terms? Are the data accurately recorded? Is the sample correctly chosen? Are the statistical techniques used appropriately?) But long before the sample is drawn or any data are collected, the investigator has, whether he has thought about it or not, made two other decisions that will decisively shape his conclusions. The first has to do with the way that the construct that is the “real” dependent variable is translated into measures, the second with the unit of aggregation that will be used to describe the effect. [3.238.87.31] Project MUSE (2024-03-28 21:59 GMT) [ 154 ] toward the best of all possible worlds operationalizing the construct Almost nothing having to do with social policy that we can measure directly is the construct we are really interested in. This has been a leitmotif throughout this book, but it is time to be more explicit. When we measure poverty, we are not really interested in whether the person in question has an income above or below the number of dollars that happens to represent the poverty line that year...