Higher Ground: New Hope for the Working Poor and Their Children

Chapter 4: The Evaluation

Chapter 4 The Evaluation T he random-assignment lottery method of selecting participants was critical in measuring New Hope’s impacts. Random assignment involved recruiting twice as many people as the program could afford and then, in effect, flipping a coin to determine who became eligible for benefits and who was assigned to the control group. Random assignment is the core method for medical trials and laboratory experiments in the sciences. Data collected from random-assignment studies are revered by most social scientists and policy experts—“nectar of the gods,” as National Advisory Board member Rob Hollister put it—most of whom have no choice but to conduct their own research using other, less convincing methods. Tracking the fortunes of the people selected for the program and those in the control group makes it possible to see how much better or worse off participants and their families were, relative to a virtually identical group of individuals who were not offered New Hope benefits but lived in the same communities, had the same levels of motivation, enjoyed the same booming economy, shared the same work ethic, and were subject to the same changes in welfare rules and poverty policies. It is not enough to find, for example, that New Hope participants were working more or enjoying higher living standards at the end of three years than when they signed up for the program, since economic conditions or the aggressive welfare policies put in place by the state of Wisconsin, rather than New Hope, could have produced these changes.1 What counts in assessing the success of a program in its particular time and place is whether its participants did better than a similar group of people who were not offered its benefits. This method of evaluation requires that New Hope’s impacts stand out over and above the variability in people’s work lives stemming from their job circumstances, family needs, and life crises. For Julie Kerksick and her staff, random assignment was a frustrating requirement of the evaluation. It meant marketing the program to hundreds of people and then telling half of them, after they had been at- THE EVALUATION tracted by New Hope’s offer of a better life, “Sorry, you lose.” Kerksick recognized, however, the importance of a random-assignment evaluation in policy making. Tom Brock described how they explained the lottery to participants and staff: We spoke about the need to learn whether or not New Hope was effective in helping people to work and lift themselves out of poverty and explained why random assignment was the best means of making this determination. If we could show New Hope produced positive effects, we said that policy makers in Wisconsin and other parts of the country might be persuaded to offer New Hope to many more people. . . . I think the surveys and qualitative interviews that were done later confirm that most individuals in the control group moved on with their lives and did not hold a grudge against the program or the study. My own sense is that there was a lot of uncertainty in the environment anyway because of welfare reform, and most people who applied to New Hope understood that they were taking a chance and had little to lose by going through random assignment. People sometimes question the ethics of random assignment because one-half of the applicants are deprived of the potential benefits of the program. It was important, of course, to inform applicants from the very beginning that they had only a fifty-fifty chance of getting into the program . Moreover, despite slower than expected recruitment, some people would have been turned away because there were funds for only about 600 participants. Random assignment was a fair method of allocating scarce slots. In a larger sense, not only is random assignment ethical, but it would also be unethical not to assess New Hope’s impacts before promoting a program for widespread use. Furthermore, the control group was not deprived of any programs or services otherwise available in the community. These considerations did not allay the anger and suspicion of some applicants and the resignation of others when their lottery numbers put them in the control group. One woman was convinced that she had been turned down because she had no children after her sister, who did have children, had been selected. Still others had years of experience dealing with the exploitation and red tape of employers, contractors, and social service agencies and were...