Using the Delphi and Snow Card Techniques to Build Consensus Among Diverse Community and Academic Stakeholders

Abstract

Background: The New York University– New York City Health and Hospitals Corporation (NYU-HHC) Clinical and Translational Science Institute (CTSI) used a community-based participatory research (CBPR) and consensus-building approach among its community advisory board (CAB) and steering committee (SC) members to formulate research priorities to foster shared research collaborations.

Methods: The Delphi technique is a methodology used to generate consensus from diverse perspectives and organizational agendas through a multi-method, iterative approach to collecting data. A series of on-line surveys was conducted with CAB members to identify health and research priorities from the community perspective. Subsequently, CAB and SC members were brought together and the snow card approach was utilized to narrow to two priority areas for shared research collaborations.

Results: Cardiovascular disease (CVD)/obesity and mental health were identified as health disparity areas for shared research collaborations within a social determinants framework. In response, two workgroups were formed with leadership provided by three co-chairs representing the three constituents of the NYU-HHC CTSI: NYU faculty, HHC providers, and community leaders

Conclusions: The Delphi approach fostered ownership and engagement with community partners because it was an iterative process that required stakeholders’ input into decision making. The snow card technique allowed for organizing of a large number of discrete ideas. Results have helped to inform the overall CTSI research agenda by defining action steps, and setting an organizing framework to tackle two health disparity areas. The process helped ensure that NYUHHC CTSI research and community engagement strategies are congruent with community priorities.

In 2009, NYU and HHC received the National Institutes for Health CTSI award. The NYU-HHC CTSI is a partnership between NYU and HHC, the largest public health care system in the country serving more than a million individuals from vulnerable populations. Designed to enhance the quality and productivity of the research effort at NYU, HHC, and across the nation, the CTSI has as its mission to promote the development of clinical and translational research at NYU using an interdisciplinary and transdisciplinary approach.

Recognizing community engagement as a critical piece of the translational pipeline, the CTSI developed the Community Engagement and Population Health Research (CEPHR) core. CEPHR is a partnership of community members, researchers, health and social service providers, community-based organizations, academic investigators, and policymakers working to advance research that identifies and disseminates relevant and effective strategies to make people and populations healthier and to reduce and eliminate health inequities.

CEPHR is guided by a CAB and an interdisciplinary SC. In 2010, CEPHR used a CBPR and consensus-building approach among its CAB and SC members to identify community-defined priorities and areas for collaboration as strategies to reduce disparities in New York City. The process took place over a 1-year period and consisted of the following multi-pronged approach: (1) Delphi technique with CAB members; (2) strategic planning process with SC members; (3) joint meeting and snow card approach; and (4) survey of capacity and resources. The result of this participatory process was the creation of two workgroups composed of community, academic, and HHC representatives focusing on CVD/obesity and mental health.

The Delphi technique is a method used to generate consensus from diverse perspectives and organizational agendas through a multi-method and iterative approach of collecting data. The technique was first developed in the 1950s by the Rand Corporation in a study that reasoned that, “one gets closer to the truth when there is the combined judgment of a large number of people.”¹ It has been used in many different fields to establish priorities.¹ In particular, it has been widely used to define research priorities, such as in occupational medicine,² occupational health research,³ and nursing.⁴ The process involves a series of questionnaires. The first questionnaire consists of one or two broad questions. The responses are then analyzed, and from these a second questionnaire is developed. In the second questionnaire, participants are asked to answer more specific questions to clarify their first responses. The Delphi technique proceeds in this manner until consensus is reached, usually requiring between two and five rounds.

The snow card technique is a nominal group technique used when performing a strengths, weaknesses, opportunities, and challenges analysis in a strategic planning process.⁵ The title “snow card” refers to the index cards used in the process. This method combines brainstorming—which produces a long list of discrete ideas—with a synthesizing step, in which ideas are grouped into thematic categories. Through this method, the diverse ideas of a group of participants can be integrated to represent agreement on a particular issue.⁵

CEPHR elected to use the Delphi and snow card techniques based on a review of the literature on participatory decision-making techniques and discussions with CAB and SC members. Important factors considered when selecting the Delphi technique included being able to systematically gather the opinion of each CAB member, and then provide CAB members with an opportunity to react to the opinions expressed by others. As a result, findings were likely to be more comprehensive than those obtained in an in-person meeting.^4,6 In addition, the Delphi exercise was completed on-line, allowing the group to more efficiently use time at quarterly face-to-face CAB meetings.

Important factors considered when selecting the snow card technique included the fact that the technique allowed for equality of participation, and generation of a high quantity of ideas compared with an unstructured group discussion.^7,8 Taken together, both methods provided the team with effective tools for identifying shared goals while attending to key principles of CBPR, such as ensuring the inclusion of partners’ different values, priorities, and expertise.

Consensus-building methods such as the Delphi and nominal group technique have been used extensively in community– university partnerships.^9,10 For example, the Delphi method has been used in CBPR as early as 1980 in a Canadian project that sought to identify key health issues in a Toronto community.¹¹ Likewise, Israel et al.,⁸ in their seminal textbook on methods in CBPR, highlight the nominal group technique as a key technique for effective group process in CBPR partnerships.¹²

Methods

Community and Academic Partners

The CAB is composed of community leaders representing a diverse cross-section of New York City’s racial and ethnic communities, government, healthcare concerns, social service agencies, and neighborhoods/boroughs. CEPHR leadership strategically invited trusted community leaders, known through existing partnerships, to serve on the CAB. In addition, CAB members were tasked with identifying new recruits with an effort to create a board representative of the racial and ethnic communities of New York City, with diverse knowledge on prevalent health issues, and a commitment to addressing health through a holistic paradigm. The group is co-chaired by Dr. Rosa Gil, Founder and President of Comunilife, Inc., and Dr. Ruth Browne, Chief Executive Officer of the Arthur Ashe Institute for Urban Health. At the time of implementing the process described in this paper, the CAB was composed of 22 members.

CEPHR is also guided by an interdisciplinary SC composed of 21 members representing NYU Schools of Medicine, Nursing, Education, Public Administration, and Dentistry as well as representation from HHC. Table 1 shows the diversity of representation of CAB and SC members by population group and geographic and research area, as self-reported in the Survey of Capacity and Resources described below.

Delphi Technique With CAB Members

A series of on-line Delphi questionnaires were conducted with CAB members to achieve the following aims: (1) To identify the health priority areas perceived as most important by CAB members, (2) to identify health priority areas that should be given special attention within the CTSI, (3) to clarify the reasons for these choices, (4) to recommend research to be undertaken by the CTSI to address the identified priority areas, and (5) to identify shared priority areas among CAB members as areas of potential collaboration.

Questionnaires were created using SurveyMonkey, a web-based survey provider, and are included in Appendix A. The two-round Delphi process occurred over a 4-week period in March 2010. During the first round, respondents were asked to respond to the questions based on their professional and personal experience and were given permission to seek the consensus opinion of their agency before answering. The first questionnaire was divided into two parts. The first asked CAB members to list the 10 most important health priority areas for the communities they serve and to explain their answers. The second part asked CAB members to list the 10 recommended research activities to be undertaken to act on the health priorities they identified and to explain their answers.

Table 1.

Areas of Expertise Represented by CAB and SC Members

Answers listed for health priority areas were reviewed by the first author of this paper and a second NYU staff researcher who identified commonalities, grouped responses into categories, and resolved discrepancies. Twenty-one health priority areas were identified and presented back to CAB members in the second Delphi questionnaire in order of decreasing priority as indicated by the frequency of citation during the first round. During the second round, respondents were asked to reflect on the combined responses of the group and to choose 10 health priority areas and indicate its level of priority (using a 5-point Likert scale from 1 [very low priority] to 5 [very high priority]). Data were entered and analyzed using SPSS Version 19.0 (SPSS, Inc., Chicago, IL). Quantitative analysis included calculating the frequency of selection for each item. Frequencies were determined for responses that included the ratings: Very high priority, high priority, and moderate priority. Results from open-ended questions on reasons for choice and recommended research activities were reviewed by the first author of this paper and a second NYU staff researcher, themes and concepts identified and coded, and discrepancies resolved. Findings from the Delphi questionnaires were presented back to CAB members for discussion at a quarterly meeting.

Strategic Planning Process With SC Members

Simultaneous to the Delphi approach with the CAB, CEPHR engaged the SC in a two-part planning retreat in May 2010. An outside consultant was hired to facilitate the retreat with the overall goals of defining the role and charge of the SC, identifying opportunities and challenges for community engagement and population health research, and developing a strategic plan.

The consultant posed the following visionary question to members: “What does CEPHR need to accomplish within the next 5 years to achieve great success?” She then asked the group to brainstorm what gaps must be addressed to achieve these goals, what resources are currently in place to achieve them, and what roles members can play to contribute to its success. Responses to these questions served as the basis for a draft vision statement, which was revisited and further refined in the second retreat, and approved through consensus.

Joint Meeting and Snow Card Approach

In December 2010, CEPHR convened a joint meeting of the CAB and SC. At this meeting, CEPHR co-directors and CAB co-chairs presented results of the CAB’s Delphi exercise and the SC’s planning retreat, and led participants through the snow card exercise described in further detail below with the purpose of identifying common goals and areas for shared community–academic collaborations.

Figure 1.

Results of the First Delphi Questionnaire (N=15)

Four large sheets of paper were taped up around the room with the following headings: (1) population group, (2) health condition, (3) geographic area, and (4) capacity building. Participants were asked to reflect on the following question, “What are some shared areas of collaboration within each of these four categories?” and to write down their ideas on index cards and tape them up under the corresponding category. Participants were then divided up into four smaller groups by counting off, with equal numbers of CAB and SC members. The smaller groups were assigned to one of the four categories and asked to move together to that category to work as a team to organize the index cards into a meaningful thematic order. Next, a volunteer from each group reported back to the larger group on the main themes for their category. CEPHR co-directors and CAB co-chairs then led the entire group in a discussion about the potential for various collaborations across categories and summarized the main discussion ideas on a new large sheet of paper. During this exercise, the facilitators encouraged members to keep in mind the level of effort needed, and the potential impact of each idea, with the ultimate goal of finding areas of collaboration that would have a high impact and require low effort.

Survey of Capacity and Resources

In follow-up to the joint meeting, CAB and SC members (n = 43) were sent an on-line survey to inventory areas of expertise in the following areas: population, geography, and research activities/methods. This survey also asked respondents to select two areas for collaboration out of the seven major areas identified at the joint meeting.

Results

Delphi Technique

All 22 CAB members were invited to take part in the survey, of whom 15 (68%) completed the first round, and 15 (68%) completed the second round. They identified 21 health priority areas in the first questionnaire. The top six areas identified were: (1) diabetes, (2) sexual health, (3) nutrition and obesity, (4) access, (5) cardiovascular health, and (6) mental health (Figure 1). The criteria used to explain the choice of a priority was the frequency of the health problem and its impact on underserved communities, using phrases such as “these diseases are increasing, particularly in the most vulnerable communities.” In addition, the complex interplay between priorities was used to explain choices, using phrases such as “obesity and lifestyle are leading to a higher incidence of diabetes and other problems.” Identifying mental health as a key priority was supported by citing the role of stigma and contextual factors such as “immigration-related stress,” “social isolation,” and “discrimination.”

Figure 2.

Results of the Second Delphi Questionnaire (N=15)

During the second round, participants were asked to reflect on the combined responses of the group before making their selection. A slight shifting of priorities is seen in the second round. For example, health education/behavioral health was identified as a priority by 3 CAB members on the first questionnaire, and 14 CAB members on the second questionnaire. However, overall there was a high convergence of opinion between the first and second rounds. The top six areas were (1) health education/behavioral health, (2) diabetes, (3) nutrition and obesity, (4) access, (5) cardiovascular health, and (6) mental health (Figure 2 and Table 2).

The recommended research activities included observational and experimental research using methods and approaches including CBPR, community engagement, practice-based research, cost-effectiveness analysis, behavioral health intervention research, and policy analysis. Research that examines the context within which people live was a major theme, as was the theme of cultural and linguistic appropriateness of research activities and health interventions. Table 3 presents illustrative quotes for these themes.

After reviewing the findings at a quarterly CAB meeting, it was agreed that next steps would include sharing the results with the SC and engaging in a participatory exercise with them to identify shared areas for research collaborations.

Strategic Planning Process With the SC

As a result of the retreat, the SC identified the following goals to facilitate and support community-engaged and population health research: (1) Provide consultation, training, and education, (2) support NYU and HHC researchers to secure extramural funding, (3) make contributions to general knowledge and to policy, (4) develop and support infrastructure, (5) advance the science of community engaged and population health research, and (6) support and foster equitable community–university partnerships. By creating an opportunity for SC members to jointly agree on goals and to brainstorm strategies to achieve them, the retreat secured buy-in and encouraged members to articulate their role in advancing CEPHR’s aims. The retreat also prompted requests for opportunities to meet and interact with CAB members, and laid the groundwork for robust faculty participation in the workgroups.

Joint Meeting and Snow Card Approach

The main themes identified under each of the four categories are displayed in Table 4. Based on the group discussion, seven major areas for collaboration were identified: (1) Mental health (e.g., the context of stress), (2) CVD and obesity, (3) policy change (e.g., training on public policy), (4) capacity (e.g., a CBPR training program for faculty), (5) prevention, (6) access (e.g., a street-to-center care model), and (7) structure (e.g., shared data sets).

Table 2.

Consensus on Health Priority Areas (N=15)

Table 3.

Select Quotes Illustrating Major Themes of Fostering Research Activities That Examine the Context Within Which People Live, and That are Culturally and Linguistically Appropriate

During the discussion, it was recommended that the group identify areas for collaboration in the context of the group’s current resources and capacities with the goal of identifying areas of high need. The goal of addressing the social determinants of health in any collaborative work framed much of the discussion. As a specific example of a collaborative project, the group discussed the idea of creating a street-to-center care model to address the high suicide rates of Latina adolescents. This initiated a broader discussion about the overarching issue of access to care, and the importance of working outside the traditional medical model to deliver care to underserved populations, for example through community health workers and community-based organizations.

The group agreed that the next step would be to inventory the capacities and resources in the room, and to form work groups around shared interests

Survey of Capacity and Resources

Eleven CAB and 12 SC members completed the survey (54% response rate). Table 1 displays areas of expertise identified for population groups, geographic area, and research-related activities. These findings informed the development of a Resource Guide for use by workgroup members. The top two choices for workgroups selected were CVD/obesity, and mental health. In response, two workgroups were formed with leadership provided by three co-chairs representing the three constituents of the CTSI: NYU, HHC, and community. CEPHR staff played an important role in providing logistical support for workgroup functioning.

Discussion

Table 4.

Major Themes Identified Using the Snow Card Approach Under the 4 Categories of Capacity Building, Geography, Population Group, and Health Area

Lessons Learned

It was important to initially undertake separate processes with the CAB and SC to allow the groups to solidify as distinct committees. First employing the Delphi technique with the CAB ensured that initial health and research priorities reflected community concerns. At the first joint meeting of the CAB and SC, many members were meeting each other for the first time so it was important to employ a participatory decision-making process—the snow card approach—that served as part ice breaker and effective decision-making tool. In addition, the snow card approach ensured that individuals first had time to reflect and write down their ideas, and ensured that individuals who may be uncomfortable speaking in large groups were given the opportunity to have their ideas included in the discussion.⁹ In addition, the approach ensured that a more vocal member could not put forth an initial suggestion that would have led the group to prematurely focus on a single idea.⁹ The final survey, which inventoried areas of expertise and polled members on final workgroup selection, was responsive to the next steps suggested at the joint meeting, and allowed for a final and systematic vote. Electing three co-chairs that represented the three constituencies of the CTSI ensured that all three perspectives and priorities would be built into the leadership of the workgroups.

It was also important that CEPHR staff provide logistical support of workgroups. It was reiterated several times that workgroups members did not have the time or capacity to take on any logistical responsibilities. This statement speaks to the capacity building required to support this type of work, including supporting community members and their organizations to allow them to participate. It was also important that workgroups be given time for building relationships. Although the CVD/obesity workgroup moved forward quickly to pursue CTSI pilot funding in its first year, the mental health workgroup made a decision to postpone submission of a proposal in recognition that such a step would be premature in terms of the need for relationship building. The mental health workgroup took additional time to identify experts in the field of mental health to bring to the table, as well as taking time to build trust among workgroup members. In addition, although the techniques described in this paper helped CAB and SC members to come to a consensus on health priority areas, there has not been a similar structured process within workgroups to identify specific projects. It has taken more time for the mental health workgroup to come to consensus among its members on a specific project to pursue.

The Delphi and snow card approaches can be more effective than a conventional discussion group process for making group decisions.⁸ The Delphi approach fostered ownership and engagement with community partners because it was an iterative process that required stakeholder input into decision making. The first questionnaire allowed for idea generation and the second questionnaire allowed for prioritizing. The approach also allowed us to gather the opinion of each CAB member, and then provide CAB members with an opportunity to react to the opinions expressed by others. By doing this, the group converged toward consensus without bias toward certain views or dominance by certain persons that can occur face to face. Employing the Delphi process also allowed us to overcome time and location constraints. The rankings of health priority areas shifted slightly as a result of the two-step process. For example, health education/behavioral health was identified as a priority by only 3 CAB members on the first questionnaire, but 14 CAB members on the second questionnaire, revealing the value of the Delphi technique in bringing forward important concepts that may originate from only one or two participants. There was however, high convergence of opinion observed between the first and second questionnaires including cardiovascular health, mental health, nutrition and obesity, diabetes, and access, underscoring the importance of these health priority areas from the perspective of CAB members

A limitation often cited in the Delphi literature is the lack of group interaction.⁸ It was therefore important that the results of the Delphi be presented back to CAB members during an in-person meeting for the opportunity to receive feedback and discuss next steps. In addition, it was important that the Delphi exercise be followed by a joint in-person meeting and use of the snow card approach. This allowed for the stimulation of interpersonal relationships and group cohesion around the common goal of identifying areas for shared collaboration. Employing the snow card approach also allowed for equality of participation among members, ensuring that each individuals’ ideas were included as part of the larger discussion.⁸ Such nominal group techniques have been shown in studies to produce more ideas and achieve greater satisfaction among participants.⁹

The multistep process described in this paper allowed us to play on the strengths of both decision making techniques, while overcoming the limitations inherent in each technique when used on its own. Results have helped to inform the overall CTSI research agenda by defining action steps, and setting an organizing framework to tackle two health disparity areas, namely, CVD/obesity and mental health. With the support of a 2012 CTSI pilot award, the CVD/obesity workgroup is currently collecting formative data on workplace interventions to address obesity and physical activity in nonprofit settings in New York City. The mental health workgroup hosted a seminar in March 2012 bringing together local academic and government experts on Latino mental health. The mental health workgroup is also currently developing a proposal for the 2013 CTSI pilot award to adapt an evidence-based intervention in community settings to address Latino mental health disparities.

Limitations

Study limitations include low response rates for the Delphi and Capacity and Resources surveys. Thus, results may not be representative of all CAB and SC members. Owing to time constraints of CAB and SC members, the tasks of data collection and analysis were the sole responsibility of CEPHR staff. However, CAB members provided feedback on the results of the two-round Delphi and discussed next steps, including plans for the joint meeting. In addition, as authors of this manuscript, the CAB co-chairs have played a key role in generating the conclusions presented in this manuscript.

Conclusion

Our aim was to develop and implement an appropriate process for identifying community-defined priorities and areas for collaboration as strategies to reduce disparities in New York City. Utilizing a multistep participatory decision making process with CAB and SC members helped to ensure that CTSI research and community engagement strategies are congruent with community priorities. The structured and systematic approaches of both techniques ensured that the diverse perspectives and organizational agendas of each member were a part of the group’s frame of reference. Participatory decision making is a cornerstone of CBPR and key to effective community–university partnerships. The process described in this paper can serve as a model for other CTSAs and community–academic partnerships interested in identifying health and research priorities to foster shared research collaborations