External Community Review Committee: A New Strategy for Engaging Community Stakeholders in Research Funding Decisions

Abstract

Background: Major gaps exist between what we know and what we do in clinical practice and community health programs and narrowing this gap will require substantive partnerships between academic researchers and the communities they serve.

Objectives: We describe a research pilot award program that makes a unique commitment to community engagement through the addition of an External Community Review Committee to the typical research review process that gives external stakeholders decision-making power over research funding.

Methods: Whereas engaging community reviewers in discussion and rating of research proposals is not novel, the ICTR ECRC review process is distinct in that it is subsequent to peer review and uses different criteria and methodology. This method of engagement allows for the community review panel to re-rank scientifically meritorious proposals—such that proposals funded do not necessarily follow the rank order from scientific peer review. The approach taken by UW ICTR differs from those discussed in the literature that present a model of community-academic co-review.

Results: This article provides guidance for others interested in this model of community engagement and reviews insights gained during the evolution of this strategy; including how we addressed conflict, how the committee was able to change the pilot award program over time, and individual roles that were crucial to the success of this approach.

Conclusions: The advantages of this approach include success through traditional academic metrics while achieving an innovative shared-power mechanism for community engagement which we believe is critical for narrowing the gap between knowledge and practice.

Traditionally, the scientific community makes decisions regarding the investment of research dollars. In addition, dissemination of results in peer-reviewed journals and conferences often bypass community stakeholders.¹ As a result, community input into research priorities is negligible. Yet the public and funders expect greater returns on their research investments.² Community-academic partnerships show promise in addressing these issues and achieving health improvements.^3–7

In 2007, the University of Wisconsin–Madison was awarded a Clinical and Translational Science Award from the National Institutes of Health (NIH), requiring us to create pilot award programs to provide researchers funding for conducting translational research. In designing its pilot program, the UW-Madison Institute for Clinical and Translational Research (UW ICTR) made a unique commitment to engagement of community stakeholders in the application process by creating an External Community Review Committee (ECRC) with decision-making power regarding expenditure of the Institute’s research funds for “Type 2” (i.e., practice-based or community-engaged) translational research (in addition to requiring applicants to—at a minimum—involve a representative of their stakeholder community in their research project). This creation of the ECRC marked a departure from typical approaches to engagement by moving beyond community engagement in an advisory role⁸ to one that shares decision-making power. Our search of the literature did not reveal the existence of other processes similar to the ECRC, although the Canadian Institutes of Health Research, the Western Australian Health Department, and the Cancer Council NSW have consumer community members review their research grant applications (members do not have a voting/funding decision role),^8,9 and we did find recommendations within the literature for developing grant application and review processes that incorporate the values of community-based participatory research (CBPR).^6,7 Our goal in creating and engaging this committee comprised of community stakeholders is to bridge the “lack of fit between the dynamics of true community collaborations and the peer-reviewed funding approach to setting research priorities.”¹⁰

Our community-academic partnership is represented fully in the writing of this manuscript through the voices of Dr. Smith (the faculty program director), Ms. Kaufman (an ECRC member and former public health nurse and consultant focused on health issues of importance to minority and low-income women and children), and Ms. Dearlove (a staff member with experience in both academic and non-profit community organizations).

Defining “Community”

With the release of its Community Participation in Research R01 PA-08-077,¹¹ the NIH broadly defined community as “referring to target populations that may be defined by: geography; race; ethnicity; gender; sexual orientation; disability, illness, or other health condition; or to groups that have a common interest or cause, such as health or service agencies and organizations, health care or public health practitioners or providers, policy makers, or lay public groups with public health concerns.” UW ICTR adopted and implemented this inclusive definition of “community” for its pilot awards program—requiring applicants to incorporate community input into their research projects and engaging community health stakeholder input in the research review and funding decision process as members of its ECRC.

The External Community Review Committee— Formative Decisions

UW considered several community-engagement options for its review process, including initial review of proposals by community members to assess the value of the research to the Wisconsin community; incorporation of community members on the scientific review panel; and final review of proposals by community members. The latter approach was selected because it carried the most decision-making power, making the ECRC ultimately responsible for making the final funding recommendations.

Although enthusiastic about engaging community members in the pilot process, institutional leaders initially voiced concern that community members would not adequately understand the scientific merit of the proposed research and would decide to fund proposals with low scientific merit. A second concern was whether the ECRC could identify faculty or projects that would succeed in subsequent applications to NIH and other research funders. A final concern was whether community members would bring institutional biases or focus only on their own priority issues. The faculty lead of the grant program, in an “engagement motivator” role,¹² worked to address each of these concerns and promote the intrinsic value of shared control over decisions. For example, we assured institutional leaders that only proposals with an adequate level of scientific merit would be forwarded to the ECRC for their review, and developed an initial ECRC review process that tied the community member scores to those determined during the scientific merit review process (ECRC members could score within a set positive or negative range from the scientific score). Although these changes to the ECRC process helped institutional leaders be more comfortable with this approach, they were ultimately required to take a “leap of faith” that outcomes of the process would have strong value to both faculty and the community.

The Pilot Program in Practice

The UW ICTR Type 2 Translational Research pilot award process is designed to attract and award scientifically meritorious research; thus, most of the application requirements and review criteria mirror components of NIH and other federal calls for proposals. But UW ICTR also used the application process as a mechanism for shaping how academic investigators think about and conduct research that is relevant to and engages communities most likely to implement the research findings. We addressed this through (1) annual changes to the proposal requirements, pushing investigators farther each year toward the principles of community engagement, (2) the criteria set forth for the reviewers of both scientific and community review, and (3) adding a new grant program in 2009 ($200,000 for 2 years) that required more robust collaboration and shared funding with community partners.

Table 1 shows the original application requirements/process and how the influence of the ECRC (through their recommendations for changes to the pilot process each year) encouraged a new paradigm that set community engagement as a required element of funded pilot research.

Methods

How ECRC Members are Chosen and Introduced to Their Role

Significant effort went towards the composition of the ECRC. Our goals for identifying members included engaging individuals who had a basic understanding of the grant review process, primarily since they had participated in community development/implementation grant reviews themselves, and who would be able to participate in discussions about research proposals without advocating for a specific disease, condition, or approach.

Our goals for building a committee were to engage a group who would represent rural and urban voices, different disciplines, perspectives, races/ethnicities; who would be committed to this endeavor and partnership; and finally, who would feel comfortable expressing concerns and honest opinions about this approach in order to strengthen the committee’s influence over time.

To ensure membership met these goals, we built a “potential member matrix” from recommendations by academic and community members. Input was solicited from faculty who were experienced with community-collaboration in research, as well as representatives of state and local government, existing community advisory board members throughout the state who been engaged in research through CBPR processes, health practitioners, etc. Once populated, this list was shared with a core group of faculty and staff who chose the first 12-member committee; a similar process was used in subsequent years and we maintain the number of members at 8–12 each year.

Table 1.

Pilot Program: Original RFA Requirements and Changes Over Time

Since 2007, ECRC members have come from health services agencies (hospitals, federally qualified mental health centers, large and small health systems, advocacy organizations, Wisconsin Medical Society); governmental agencies (State Department of Health, State Division of Public Health, county health departments, the Oneida Nation, former legislator); nonprofit organizations (Hospice, League of Women Voters); and educational organizations (UW Extension, state technical colleges, universities).

Between 2008 and 2012, a total of 32 unduplicated members served on this committee; each year a core number of members (up to 75%) return to provide continuity and historical perspective for the process. We provide honoraria for participating members, as many have hours of travel and time away from work.

The ECRC Training Process

We deliver a training primer prior to ECRC meetings to affirm the goals of the ECRC process and pilot grant program. When a group of prospective members is selected, staff contacts each individual to introduce the mission of the grant program, the intent of the community-driven committee, and ascertain interest and availability. If interest is expressed, the prospective member is sent a copy of the current “request for applications” (RFA) as well as a briefing document defining “type 2” translational research and the UW ICTR’s pilot award review process (see Appendix A).

Because much correspondence is initiated before actual proposals are submitted, ECRC members have approximately four months before they receive proposals. Once proposals go through the first three steps of the review process (technical, peer, and study section review), selected materials from those proposals that have achieved a minimum level of merit during the scientific review part of the process are sent to ECRC members for their review in advance of the ECRC meeting.

The final step of the ECRC training process takes place at the actual meeting, before discussions begin. First, there is time allotted at the beginning of the agenda for the committee members to get acquainted or re-acquainted before the actual proposal discussions begin. The Chair then presents a PowerPoint (see Appendix B) that provides an overview of the CTSA (UW-ICTR), a review of the kinds of research that this award program supports (“type 2 translational” research), conflict of interest/confidentiality, and the tasks before the committee for that day—with ample opportunity for questions and answers. We provide coffee, snacks, and lunches to members during the day’s activities.

The ECRC Meeting

Members review but do not score components of all the scientifically meritorious proposals in advance of the meeting. These materials initially included only the scientific abstracts for review, but each year, ECRC members requested that additional sections be added to the RFA requirements—and that they subsequently receive these sections in their review portfolio. ECRC review portfolios now include a community abstract (added requirement in 2009); letters of collaboration and support from community collaborators (2009); sections of the proposal narrative that address the nature and extent of community collaborations (2009, 2010, 2012); and a section related to dissemination plans and future policy impact of research findings (2012).

While the portfolios initially presented proposals in scientific merit order, as institutional leaders grew more comfortable with the process—and at the request of the ECRC—we discontinued the approach of tying ECRC scoring to scientific merit scores and presented proposals in alphabetical order for the ECRC to rank based solely on their review criteria, analysis, and discussion.

ECRC Discussion and Scoring

After the initial presentation, the Chair presents a summary of each proposal to the group and then opens the discussion. ECRC members use the following criteria to evaluate and score proposals:

• • Significance: Does this research area address important problems or critical barriers to progress in advancing/improving clinical practice and/or community health and policy? Does this project have long-term potential to contribute to the advancement of health?

• • Priority: Should the aims of this proposal be a research priority for UW ICTR?

• • Community: Does this research project effectively and meaningfully incorporate the input of community partners/stakeholders?

Every committee member is asked to review and comment on every proposal (except in the case of a conflict of interest); therefore, there is no “lead presenter” who begins the conversation per se. Members take turns discussing the merits of each proposal based on the review criteria and make specific recommendations to the applicants when warranted (e.g., suggestion of an alternative or additional community partner, advice with regard to the feasibility of a given approach based on a member’s unique perspective of the community involved, etc.). As expected from the literature regarding effective community-academic partnerships, members want to see evidence within the proposals that the research team values community contributions and focus on community-identified issues and opportunities.¹³ Disagreements among committee members are addressed until no further discussion is warranted and then each member assigns their scores on a private score sheet (see Appendix C). PowerPoint slide deck for meeting available at See http://muse.jhu.edu/journals/progress_in_community _health_partnerships_research_education_and_action/v007/7.3.smith_supp02.pdf

All comments are captured in the meeting minutes and are shared with applicants as part of their de-identified critiques. For those who are awarded funding, they are required to submit a letter to UW ICTR responding to all review concerns (both scientific and community) before they can receive funding.

A significant change in the ECRC discussion and scoring process occurred in 2012 when the oversight and management of the committee’s proceedings was turned over to a community member. From 2007 to 2012, the faculty lead managed the ECRC meeting and clarified questions related to the mission of the committee—without participating in discussions or scoring. In 2012, the discussion was led by a new “community” chair, with no faculty members present at the ECRC meeting. This change was possible for two reasons. First, the presence of the faculty lead was always intentionally limited to a coordinator, rather than a discussion participant; therefore, the transition seemed timely and certainly justified by the committee’s mission, role, and design. Second, approximately one third to one half of the committee members had participated in every meeting since the first round of funding—and the relationship/trust built among the members, the faculty lead, and the staff allowed for amenability to this change.

Results

Over the first five rounds of the program, ICTR received 141 Type 2 Translational Research applications, and of those received, the ECRC received a total of 63 scientifically-meritorious applications to review. UW ICTR was able to fund 41 awards.

Overall, the 41 funded applicants have shown a return on investment of approximately $20 million in federal/external non-federal grants and 16 publications to date.

The ECRC intervened and changed funding decisions 21% of the time, such that 9 projects were chosen for funding because of ECRC review; in other words, they would not have been funded had it not been for the ECRC. These 9 projects have provided a substantial return on investment—including 6 funded external peer-reviewed grant submissions, for a total of almost $4.5 million, as well as 7 publications in peer-reviewed journals. This translates to the “rescued” projects representing 23% of the subsequent grant funding and 44% of the subsequent publications.

Although the sample size is too small to offer a causal explanation for their success, we know that the research programs of these PIs were highly attuned to the interests of funding agencies and represented programs for which funding was available, such as patient-centered care research and health equity/disparities research. It should also be noted that the ECRC members agreed with scientific reviewers on the remaining 32 projects—they recommended that they be funded as well.

For those academic health centers or clinical and translational science award recipients who may be interested in using this model of community engagement, we would like to share our own insights from this process that may provide value to those considering replicating this model of engagement.

1. Communicate openly and consistently about roles and expectations; and anticipate some conflict

Each year, conflicts arose, resulting from different priorities, perspectives, values, and language. We had extensive discussions about issues crucial to meaningful scoring, such as: how to define “community” (e.g., whether research conducted in collaboration with clinicians at academic health center clinics was a “community” partnership); how to determine “impact/significance” (e.g., by the percentage of state population affected versus severity of disease); and how to assign “priority” (e.g., individual-focused versus community-wide interventions).

We also depended on open communication in order to ensure that the review process was one of genuinely shared power. Each year that the committee asked for changes in the RFA, requested additional pieces of the proposals to make their decisions, or wanted to untether their scores from those assigned during scientific review, we discussed the pros and cons of these changes together, and then for significant changes, the faculty lead made the case to ICTR administration to modify our methods and codify these changes.

Feedback from community reviewers was synthesized and considered after every review cycle. UW ICTR’s willingness to accept changes to the process recommended by the ECRC created opportunities to improve the application and review processes in ways that strengthened both the funding program and the relationship between academia and research in the community and ultimately, we believe, resulted in stronger applications. (See Table 2 for quotes from ECRC members regarding their appreciation of how the pilot process changed to reflect their concerns and feedback.)

Table 2.

ECRC Commentary

2. Expect changes—from both academia and community members—to be incremental and be prepared for the time commitment

Recognizing the evolution of the role of the ECRC over time is important for those considering implementing this model. UW initially expressed concerns about creating a review component that gave funding decision-making authority to a group of non-academic community health stakeholders after scientific review. Community members were at first concerned that the process would not truly share decision-making power, and that decisions “not liked” by institutional leaders would be reversed or ignored. Using an incremental approach over a five-year timeframe proved critical to success, allowing time for academic and community stakeholders to develop trust and comfort with engagement processes and change.

Engaging communities in making funding decisions about research requires commitment to the process and a long-term dedication to keeping eyes on the prize. Having an academic leader or “champion” who was accountable for creating, managing, and consistently advocating for these processes was vital to the program’s success. This individual must have internal respect and be capable of managing the tensions between the hierarchy of an academic research setting and expectations of community member reviewers.

3. Choose the right partners for your mission

Success in this experiment required creating defined roles and engaging people with “the right stuff” to fill them. On the academic side, commitment to community engagement and leadership by the ICTR Principal Investigator (PI) proved to be vital in making wide-sweeping changes over a short time trajectory, especially because change involved determining who received funding. The faculty lead and the ICTR PI were engagement motivators.¹² The faculty lead paid careful attention to process and system issues, communicating the goals and norms of the program clearly, assuring transparency, and exemplifying sharing of power and true collaboration.

Hiring “boundary spanners” (non-faculty staff who ran the day-to-day processes and communicated extensively with the faculty lead, research applicants, grantees, and ECRC) proved critical to our success.¹² This staff listens, interprets thoughtfully, and communicates clearly. Their passion for academic research that makes a difference in community practice clearly reduced friction and led to creative solutions for problems that arose. Both had experience in community and university settings, and could speak the language of both camps, while conveying fairness and shared power.

University assets were balanced by thoughtful and experienced community reviewers. Diversity of backgrounds and experiences in delivering and managing health services and policies were enhanced by choosing reviewers who have applied for community development/implementation grants and managed them. It was crucial that reviewers did not come to the table to advocate on behalf of any organization that they might be associated with—but instead consider the overall needs of the health of Wisconsin residents (this expectation was reiterated at each meeting). Over the five years of these meetings, no one individual overwhelmed the group’s discussion—members naturally take turns presenting their perspectives and actively engage other members in discussion. Finding committee members who are committed to this group process is invaluable.

Discussion

While we recognize that engaging community reviewers in discussion and rating of research proposals is not novel, this method of engagement differs in that it allows for a community review panel to re-rank scientifically meritorious proposals—using community engagement criteria—with the result that proposals funded do not necessarily follow the rank order from the first level of scientific peer review.

Co-review models in the literature present goals of community reviewer programs as providing mechanisms for public accountability, increasing transparency in the peer review process, providing feedback on lay abstract portions of applications, and communicating merits of investments in health research.¹⁴ These programs note, however, that “Community Reviewers do not score nor appraise the scientific quality of the applications.”⁸ NIH has also made contributions to the discussion of community engagement in research; however, these recommendations and practices refer primarily to training researchers how to effectively conduct and participate in community-engaged research and training reviewers how to best evaluate community-engaged research.⁶

While the outcomes of co-review have important implications for community-academic partnerships, they do not tackle the goal of providing shared power to community partners in academic research funding decisions. This recommendation is clear in the CBPR literature, specifically, “the input of the community participants in the review process must be heard and incorporated into the final decision-making processes. For example, community members need to be oriented to how the review process is conducted, their roles need to be clearly defined from the beginning, and how their input is going to be ``weighted” needs to be clarified. Community members’ perspectives and expertise might best be applied to assess specific partnership-related criteria across all applications, rather than taking a lead review role on the entirety of a few applications.”⁷ Each year we ask our External Community Review Committee members if they would be interested in a co-review process in which they would serve alongside scientific reviewers. Our ECRC members have consistently declined this approach, sharing that they feel this would diminish their role and their voice in the review process.

Perhaps not surprising is that the success of this approach may well be related to its design following the same recommendations found in the literature for creating successful community-academic research partnerships: the importance of personal connections, open and bidirectional communication, full participation by all parties, ability to deal with unexpected consequences, mutual benefit, and equal respect.^6,13

Engaging community reviewers as decision-makers for academic translational research grants created a successful program jointly owned by academic and community partners. Although it is still too early to determine long-term impact from grants selected by the ECRC for funding, indicators signal clear promise. The program achieved traditional academic metrics of subsequent grants and manuscripts for the pilot awardees, but also succeeded in engaging a cadre of enthusiastic community members from around the state who remain committed to the process.

The ECRC developed by UW ICTR represents a promising new strategy for engaging community members in research funding decisions. This model could be adopted or adapted by other academic funding programs wishing to increase the engagement of community members in their research processes. The advantages of this approach include success through traditional academic metrics while achieving a high level of community engagement critical for narrowing the gap between knowledge and practice.