The Fifth Dimension: An After-School Program Built on Diversity

Chapter 4

Chapter 4 Evaluating the Model Activity Systems: General Methodological Considerations valuating the Fifth Dimension is a complex challenge. In light of the goals and design of our project, our evaluation efforts have addressed these questions: • What are the effects of Fifth Dimension participation on children’s learning, and how do those effects emerge? • What are the effects of Fifth Dimension participation on undergraduates ’ learning, and how do those effects emerge? • What factors contribute to the sustainability or demise of Fifth Dimension programs? These questions are, of course, interrelated. In general, Fifth Dimension projects are more likely to be sustained if they are enjoyed by participating children and undergraduates and enhance their learning; reciprocally, enhanced learning for children and undergraduates ultimately depends on a program’s sustainability. Two considerations have framed our thinking about these questions: the family of theoretical ideas about the sociocultural situatedness of learning that has oriented our work (chapter 2) and the prevailing standards for social science analysis (Slavin 2003). From the perspective of the former, “moments” of individual learning (microgenesis), ontogeny, and cultural history, activity, and context are all intertwined (Vygotsky 1978; Wertsch 1985). This perspective has led us to conduct diagnostic evaluations in situ—that is, in local contexts and activities—with an eye on processes of change and ecological validity (Bronfenbrenner and Morris 1998; Cole 1996). Showing development in situ, however, does not meet the criteria of “scientifically based research” represented by the tradition of experimental and quasi-experimental designs, the random assignment E of children to different “treatments,” and statistical analysis of group achievement as the aggregate of measures of individual achievement (Reyna 2002). We want our evaluations to speak credibly to audiences subscribing to both traditions. These audiences include stakeholders in both the community and the university who influence local projects’ sustainability and the career paths of our professional colleagues. Consequently , our evaluation efforts have confronted theoretical and methodological tensions with a long history in the social sciences. We worked to resolve those tensions first by honoring our intellectual foundations. We conceptualized the Fifth Dimension as an activity system in which activity is analyzed over time at multiple, intertwined, contextual levels and each level is formed and influenced by the levels above and below it. Figure 4.1 portrays the system using the “embedded circles” representation of context introduced in chapter 2. We then attempted to carry out evaluations at multiple levels using a mixture of quantitative/experimental and qualitative/process data. In the sections that follow, we summarize our attempts to pursue this multimethod approach applied to data on outcomes for the children (chapters 5 and 6), for the college students (chapter 7), and for institutional sustainability (chapter 8) in ways that are both scientifically respectable and practically useful to implementers. In this chapter on general methodological issues, we begin with a cautionary tale about our efforts to carry out what we considered adequate evaluation of the impact of the Fifth Dimension on children’s academic development. Then we summarize the expanded efforts of the entire consortium of Fifth Dimension implementers to formulate more adequate solutions to the problems we failed to deal with effectively. WHEN THE BEST-LAID PLANS MEET A RECALCITRANT SOCIAL ECOLOGY When Cole and his colleagues first started to use the Fifth Dimension prototype , it was logically impossible to prove that children would benefit from spending several hours a week interacting with undergraduate students around computer-based educational activities, because the Fifth Dimension was just one of several intervention activities in a single program that focused on reading instruction (LCHC 1982). Subsequently, when they began implementing several Fifth Dimensions as self-contained after -school interventions, they had to deal with the possibility that the quality and impact of the activity might vary from one implementation to the next. For example, implementations in school-like settings might have a greater impact on academic achievement than implementation in a boys and girls club. EVALUATING THE MODEL ACTIVITY SYSTEMS 67 To address the issue of program evaluation, the research team proposed to carry out a full experimental study of the impact of the Fifth Dimension system in each of three institutions: a child care center, a BGC, and a library . The team’s high hopes for a technically convincing evaluation appear almost shockingly naive in retrospect. In the initial grant proposal, Cole wrote: All children participating in the program at each site will...