The Violence Intervention and Prevention Team (VIP):Results of an Action Research Evaluation
Background/Objectives: This paper describes a school-based youth violence prevention program and challenges encountered during efforts to evaluate it. Members of a community partnership team helped to shape the quantitative and qualitative data collection and to interpret results.
Methods: 48 youth participants in the violence prevention program completed a survey soliciting information about violence-related risk and protective factors, including employment readiness, school connectedness, association with delinquent peers, and violence-related attitudes, intentions, and behaviors. Fourteen youth also participated in two focus groups about their satisfaction with the violence prevention program.
Lessons Learned: Through the preliminary data collection process, we learned three key lessons. (1) Institutional Review Boards (IRBs) new to community-based research may need to build capacity to evaluate the human subjects implications of this type of research. (2) The identification of control or comparison groups for school-based youth violence programs may be challenging and costly. (3) Methods for reducing loss-to-follow-up with high-risk youth are needed.
Youth violence, prevention, community, gang violence
The Violence Intervention and Prevention (VIP) program is a youth violence prevention program at Haverhill High School in Haverhill, Massachusetts. To our knowledge, it is unique because, although it is school-based, it does not entail classroom learning or a curriculum. Instead, the program uses an after school club model, with strict criteria for students who may attain one of three levels of membership. The program also functions as the youth branch of the city-wide violence prevention coalition, and enables students to participate in those community meetings.
A portion of the funding for the VIP program comes from the Massachusetts Executive Office of Public Safety via a grant to reduce youth violence awarded to the police department. The grant also funds a "local action research partner" to work with the grantee team to design and implement process and outcome evaluations. "Action research" is a term for participatory program planning and research, and is considered essential for developing effective local violence prevention programs because it combines community expertise with researchers' access to and familiarity with the use of surveillance data and the evidence base.1 The action research partner for the Haverhill community is the Boston University School of Public Health. In 2007, we embarked together on a participatory process with the goal of assessing the impact of the VIP program on youth violence in the community. This paper describes the VIP program and challenges we faced in preliminary evaluation data collection, including three key lessons learned from this experience. [End Page 181]
Description of the VIP Program
The VIP program was established in 2005, subsequent to what was intended to be a one-time, peer-led violence prevention conference at the city's two middle schools organized by the school system's wellness staff (authors CI and LC). After the conference, school personnel urged the organizers to repeat it the next year. Moreover, the 16 high school student–participants in the conference asked the wellness staff to continue to meet with them on a weekly basis, and whether they could invite peers to join them at these meetings. Therefore, during the 2006–2007 academic year, the group was regularly attended by approximately 30 students, and the wellness staff and youth began to develop a mission, governance structure, and to plan activities. It was decided that the program would promote peace (i.e., reduce youth violence) in Haverhill by training high school youth to provide conflict resolution training to middle school students, educating youth about related topics such as dating violence, linking members with the community violence prevention coalition, providing academic tutoring and job placement assistance, and organizing field trips and events such as tours of local colleges. The school wellness staff also received a small grant from the state that allowed them to hire a former street outreach worker as an additional adult staff person (author AP), and to establish an in-school headquarters, which was furnished with several couches, Internet-connected computers, a dance floor, a stereo, a video game console, a plasma television, musical instruments, art supplies, and books.
The VIP program quickly developed a positive reputation within the school, and membership was sought out by 70 students during the 2007–2008 school year, many of whom were "high risk," meaning that they had a history of delinquency, [End Page 182] associated with delinquent peers, may have been gang members, were suspended for physical fighting, faced academic challenges, or were at risk for school dropout. Although some students were referred to VIP by guidance counselors or teachers, the school wellness staff and their student leadership group determined that membership would only be granted to those who were motivated to become actively engaged of their own accord. Thus, the VIP leadership developed a new three-tiered membership structure, and criteria for remaining a member (Table 1), and established the tradition of a formal induction ceremony—and nonviolence pledge—for students who were accepted as members. This new degree of selectivity only increased the visibility and desirability of program membership among students, and during the 2008–2009 academic year a waitlist was established to cap the number of members at 40 so that weekly meetings would remain productive. As of December 2009, more than 400 students have at some time been affiliated with VIP as allies, members, or leaders. The membership period has ranged from several weeks to 4 years, and most students have been members for 1 academic year or longer.
Description of the Community Partnership Team
The VIP program is affiliated with two community coalitions. First, it functions as the youth branch of the Haverhill Community Violence Prevention Coalition (HCVC), and two VIP student–leaders serve as board members of the HCVC. Second, it is supported by the grantee partnership team that was convened in response to the funding opportunity from the Massachusetts Executive Office of Public Safety. It is this second partnership team that has participated in the design and implementation of VIP evaluation.
The partnership team includes representatives from law enforcement, city schools, one community-based agency, the HCVC, and academic research partners. We meet monthly at the Haverhill police department, attend quarterly statewide meetings together, and communicate regularly via e-mail between meetings. During the past 3 years, we have worked collaboratively to define the target population for local youth violence prevention efforts, assess community needs related to youth violence prevention, set objectives, and select and implement prevention strategies to accomplish these objectives. [End Page 183] We have also explored possible methodologies for tracking programmatic activities, assessing participant satisfaction with these activities, and ultimately determining the impact of the activities on the prevalence of youth violence.
The first collaborative evaluation-related activity that we undertook was the development of a logic model (Figure 1). Next, we devised methods for tracking process data (e.g., the number of VIP participants, duration of enrollment). Finally, we began to create the tools and procedures, and identify data sources, that we would need in order to evaluate the program impact. Below, we describe our methods for and results of preliminary data collection that occurred in 2008, and then present the lessons learned from this experience.
We conducted a mixed methods investigation designed to explore (1) the characteristics of VIP on participants enrolled during the 2008–2009 academic year, and (2) their satisfaction with the program. All evaluation procedures were approved by the IRB at the Boston University School of Public Health and the principal of Haverhill High School. We assessed VIP participant characteristics using a self-report survey and program satisfaction through two focus groups with VIP participants.
The survey was developed with input from all members of the grant partnership team, included 99 questions, and took students approximately 20 minutes to complete. The survey was implemented by a trained research assistant (RA) in January 2008. The data collection procedure was as follows: VIP adult advisors informed the students about the opportunity to participate in the evaluation research 2 weeks in advance. On the appointed date, the RA met the students in the after school room, provided them with additional information about the survey, and asked for assent. She then handed out the paper surveys to students who completed and returned them to her. The students were provided with pizza as remuneration for their time. In total, 48 students completed surveys (96%). Two students were absent on the day of the data collection for reasons unrelated to the research. Respondents were 60% female, 56% Hispanic, 13% black, and 8% white (Table 2).
The survey comprised items from the following measures: The Work Opinion Questionnaire (α = 0.66−0.87),2 Attitudes Towards Gangs (α = 0.74),3 a Modified Depression Scale (α = 0.74),4 a truncated version of Rosenberg's Self-Esteem Inventory (α = 0.74−0.80),5 the Likelihood of Violence and Delinquency scale (α = 0.89),6 Friends' Delinquent Behavior (α = 0.55−0.68),7 and the Safe Dates Perpetration in Dating Relationships scale (α = 0.93).8 In addition, we assessed school bonding using 10 items from the Add Health survey,9 beliefs about conflict using 10 items from the New York City Youth Violence Survey,10 physical fighting and drug and alcohol use using 8 items from the Youth Risk Behavior Survey, delinquent [End Page 184] behavior using the SAGE survey and Teen Conflict Survey.6,11 Data were hand entered into an Access database and then analyzed using STATA (STATE, Inc., College Station, TX). A cross-sectional analysis was conducted.
The qualitative focus group questions were also developed by the members of the grant partnership team. Open-ended questions were designed to elicit information about strong and weak elements of the VIP program, how the program affected members, and whether VIP had had an impact on the school and neighborhood. The recruitment of focus group participants was conducted through a two-stage process. First, VIP adult advisors nominated 30 students as potential focus group participants, with the understanding that these students should represent a cross-section of VIP members. Next, the evaluation team randomly selected 15 of the nominees for inclusion in the focus groups, which was conveyed to them by the VIP advisors. On the day of the focus groups, 14 students were present to participate. They ranged in age from 13 to 18 years old.
Both focus groups were conducted by a master's level, trained RA. The sessions were audio-recorded and transcribed. The focus groups were conducted in a classroom at the school, during after school hours, and took approximately 1 hour. Once transcribed, the data were cleaned to remove any personally identifying information, and the transcripts were coded by hand. Codes were generated to reflect the focus group and interview questions, and additional codes were developed subsequent to an initial read-through of the transcripts. For example, an initial code was: "Strengths of the program," with a secondary subcode of "VIP is a lifestyle." Codes were applied to chunks of text (i.e., phrases or sentences) by the RA. The code application was then checked by an additional rater. Finally, coded text was grouped by theme and particularly illustrative quotations were selected for presentation. A detailed report summarizing the focus group research is available from the authors.
Of the 48 VIP participants, 6% reported that they had ever belonged to a gang, 17% reported that they had ever been arrested, 31% reported that they had ever hit or sexually assaulted a dating partner, and 23% reported that they had friends who were involved in gang activities (Table 2). A relatively small proportion of students received mostly A grades during the previous semester (23%), and only 31% reported feeling like a part of the school (Table 2). Thirty-six percent expressed doubts that they could succeed at a job (Table 2).
Focus Group Research
Several themes emerged from the qualitative research. First, students reported that they view VIP as a "lifestyle" or code of conduct rather than simply an after school club, which keeps them committed to upholding the pledge of nonviolence (Figure 2). Several reported that they achieve a level of "fame" in the community for being VIP members [End Page 185] and that younger children "look up to them," and that this is a motivator for youth to remain involved. Students also reported caring so deeply about the reputation of the VIP group that they would voluntarily quit when "in trouble," so as not to "tarnish" the reputation of other VIP members. In the words of three participants:
I am not going to lie—there are a bunch of people [in VIP] for the fame and such, because we are known so much.
Last year, I dropped [out of VIP for a while] … I did something that was kinda stupid and I didn't want to tarnish the name of VIP so I dropped it.
A lot of us [drop out temporarily] because we don't want to make VIP look like a hypocrite.
In addition, participants reported that VIP membership has taught them nonviolent ways of handling interpersonal conflict, enhanced their leadership skills, improved self-esteem, provided social support, and strengthened academic goals. For example, one participant reported:
Today … me and this other kid bumped into each other, so then he starts cursing and swearing at me. And then I looked back and this other kid from VIP was there and so we just started laughing at him and walked away. Just because of VIP I didn't get loud. I didn't swear at him. I just walked away.
Others commented on the impact of VIP on their academic goals:
I, for one, didn't even think about college because in my family money is tight. Now I am beginning to learn through VIP … that there are other ways to get to college.
[P]rior to VIP, I didn't think about college. I am going to be the first person in my whole entire family to graduate from high school. VIP has motivated me … they gave me a light.
Finally, students made suggestions for how day-to-day VIP operations could be improved, including holding a greater number of their events in school buildings as opposed to city buildings, and for adult advisors to notify students of planned events with more advanced notice.
Lessons Learned During the Evaluation Process
The evaluation was originally designed to be a pre-experimental outcome evaluation. That is, we implemented the survey described above in both January and May of 2008 with the intention of comparing the pre- and posttest data to assess changes in academic performance, work readiness, physical fighting, and other measures. However, there were at least three challenges that arose that affected our ability to collect and interpret the data.
Our first challenge was that the IRB process was lengthy and ill-suited for a one-year state grant and a community-based participatory process. The Boston University School of Public Health IRB, like those of many academic public health institutions, typically reviews human subjects applications for federal National Institutes of Health grants or for multiyear foundation awards. From start to finish, the approval process can take several months. In addition, because we wanted to collect sensitive information from youth (i.e., violence perpetration), we needed to apply for a federal certificate of confidentiality to reduce the likelihood that our data would be subpoenaed. Therefore, the timing of the process precluded us from being able to implement a pretest with students before they had been exposed to VIP during the academic year, that is, in September of 2008. Our receipt of the 12-month grant award was announced in March, and the community partnership team needed several months to work collaboratively on an evaluation design and instruments thereafter. By the time the IRB had completed their review, the academic year was already half over and the end of the grant period was in sight.
Further complicating the IRB review was the fact that the reviewers had limited prior experience with community-based research; as is common among Association of Schools of Public Health IRBs, they were more familiar with clinical trials or hospital-based intervention research.12 As a result, it was not immediately evident to either the IRB staff or the academic research partner whether the VIP advisors who were facilitating data collection were "engaged in research," and thus needed to apply for their own Federal Wide Assurance, or whether the research team would be obligated to intervene with the youth if they reported illegal behaviors such as dating violence perpetration. In the end, it was determined that although the VIP advisors were identifying students for participation in the research and keeping the master list that linked their name and study ID number, that this was not the same as recruiting, determining eligibility, or enrolling them, and therefore they were not engaged in research. Further, it [End Page 186] was determined that because the research team would not have the students' personally identifying information, they would not be obligated to report their illegal behavior to law enforcement.
Our second challenge was related to the identification of a control or comparison group. Ideally, outcome evaluation research utilizes a quasi-experimental or experimental design, where program participants' progress is measured against other students who do not participate in the program being evaluated. There were three primary reasons why we were unable to identify a control or comparison group for this evaluation. First, identifying a comparable group of students within Haverhill High School who are not exposed to the VIP program would have been challenging at best. "High-risk" students who are not involved with VIP are likely to be school dropouts, have low attendance, and to be otherwise difficult to recruit into evaluation research studies. Students waitlisted for VIP are not comparable with nonparticipants because there is substantial contamination—that is, they are incidentally exposed to many of the public VIP events and typically associate with VIP-involved peers. Second, randomizing students to receive VIP or not was out of the question—the premise of the VIP program is that it is voluntary. Third, identifying another school system in a comparable city where students have no exposure to VIP or a similar program would have taken considerable resources; meeting with school principals to present a research idea, particularly when the idea is that their students receive no intervention, is an involved process. Given that the evaluation research budget was limited and that we had 12 months to complete the evaluation, the identification of a comparison or control group was not feasible.
Our third challenge was that it was difficult to follow up with students. We did implement our quantitative survey at two time points with the intention of comparing outcomes across them, even though the "baseline" or pretest data were collected halfway through the academic year (January 2008). The follow-up took place in May 2008. Although we had a 96% VIP member participation rate for the baseline survey, the posttest had a response rate of 48%. This was determined to be insufficient to make meaningful comparisons of pre- and posttest data, so we have relied on the baseline results alone to characterize the VIP student sample. We believe that the reasons for the low follow-up response rate are twofold: (1) Students are busier than average in May because it is near the end of the school year, and therefore more likely to be absent from VIP meetings where data collection occurred, and (2) in Haverhill, high-risk students are often absent on warm days in the spring semester, and our follow-up data collection took place on such a day. In the future, we would plan to reschedule our follow-up data collection for a day when more students were likely to be in attendance.
We learned several important things from this participatory evaluation process. First, we gained new information about VIP participants, including confirmation that a substantial proportion were prior gang members, had ever been arrested, had friends who were gang members, or had been in a physical fight in the past year. In addition, we learned that they needed both academic and job readiness support in order to increase the likelihood that they would pursue postsecondary education and employment, which are both protective for violence perpetration.13–16
The finding that 38% of VIP members are male caused us to think critically about the gender composition of VIP, given that the proportion of gang members who are male is estimated to be 62% to 94%.17 One important consideration is that the requirement that members attend weekly meeting prevents some males who are on sports teams from joining. To resolve this problem, the male adult VIP advisor also leads a young men's discussion group in the school, which is open to both VIP-affiliated and nonaffiliated male students. Moreover, because VIP is designed to be a peer-led prevention program, the adult advisors have observed that socially connected female members can be effective at reaching male peers who are at risk. In other words, some males who may be at risk for violence may not join VIP, but will attend the dances and other public events, and receive exposure to the VIP antiviolence messages, in part because of the female VIP members. Therefore, the gender composition of VIP is not currently considered a major weakness of the program. We also obtained qualitative feedback data from the student participants in VIP that suggested that they were highly satisfied with the program overall, that they particularly enjoyed the notoriety associated with being a VIP affiliate, and that they had some minor suggestions for improving the day-to-day logistics of VIP operations. [End Page 187]
As important as the evaluation results themselves, we learned that we face several challenges as a community partnership team attempting to undertake research. As has been noted elsewhere, relatively few community-based participatory research studies have managed to produce strong collaborations and also employ rigorous methods.18 Our challenges stemmed from several sources, which included our relatively small budget, the brevity of the grant period, the complexities of identifying and recruiting comparison groups of students, the relative inexperience of our IRB with community-based participatory research proposals, and the difficulty of following up with harder to reach adolescents. We have also realized that including youth in the development and implementation of evaluation research will likely have important benefits. With the lessons learned from our first attempt to collect outcome data to assess the impact of the VIP program, we are now prepared to implement a pre-experimental evaluation that will yield informative results. In the meantime, we have utilized the process data that were collected to make improvements to VIP operations (e.g., more timely communication to students about events), and to characterize VIP participants for our community partners and funders.
The authors thank all of the current Charles E. Shannon grant Steering Committee members for their contribution to this project, including Thomas Fram, Abigail Isaacson, Michael Lahey, Katherine Lavigne, Megan Shea, Sean Scharneck, Linda Soucy, and Michael Wrenn; and James Stark of the Massachusetts Executive Office of Public Safety for his contributions to the initial draft of this manuscript.