A Realist Approach to Analysis in a Participatory Research Project

Abstract

Background: Realist reviews have shown the effectiveness of participatory action research but the realist approach has not been used in combination with a participatory approach in qualitative data analysis.

Objectives: To study the links between preexisting conditions in neighborhoods and the kind of actions taken at the community level during the coronavirus disease 2019 pandemic in Toronto, a community–university research partnership used a critical realist approach to analyze qualitative interviews with grassroots leaders. This article describes the procedures developed to enable participation of the full community– academic team in the analysis.

Methods: One analyst coded paragraphs in all 46 interviews for preexisting conditions (contexts), actions taken (intervention components), the often implicit factors that underpinned the actions (mechanisms), and observed results (outcomes) as stated by the interviewees. Each interview was summarized in terms of the contexts (C), actions (I), mechanisms (M) and outcomes (O) identified and one to seven midrange CIMO hypotheses were developed for each interview. A second level of analysis involved sense-making workshops with the community partner and a cross-section of interviewees using the CIMO statements.

Conclusions: This article describes the realist approach to analysis and the changes that were made to enable a mixed team of community leaders and academics to generate overall statements of impact. This is a novel approach to qualitative data analysis, with a range of implications for the use of this technique in participatory research.

To study the links between preexisting conditions in neighborhoods and the kind of actions taken at the community level during the coronavirus disease 2020 (COVID-19) pandemic in Toronto, a community organization– university research partnership used a critical realist approach to analyze qualitative interviews with grass-roots leaders. This article describes the procedures developed to enable participation of the full community–academic team in the analysis.

A realist approach to research asks the question, “What works for whom under what circumstances?”¹ Realist evaluation was developed to understand why some interventions work well in one organization or neighborhood and not another, recognizing that social problems and settings are complex.² “To understand how an intervention might generate different outcomes in different circumstances, realism introduces the concept of mechanisms, which may be helpfully conceptualized as underlying changes in the reasoning and behavior of participants who are triggered in particular contexts.”² This approach is theory-driven in proposing why an intervention might work in one context and not another and then “testing” this theory either through analysis of the literature (most common) or data generated by research. At the core is the need to identify the context (C), mechanisms (M) and outcomes (O). The context consists of the individual, interpersonal, organizational and other features of the setting in which the program or intervention operates.³ Mechanisms are the underlying (often hidden or implied) processes, forces, or interactions that lead to or inhibit change,³ and outcomes are the results generated by certain mechanisms operating in certain contexts. A realist approach assumes that there is a social reality that cannot be measured directly but can be approximated in the form of a program theory of change— what works for whom under what circumstances?^1,2 It is the relationships between the contexts (C), mechanisms (M), and outcomes (O) that are of interest. The mechanisms operate in (and are affected by) particular contexts to generate outcomes of interest.² The relationships between the contexts, mechanisms, and outcomes (CMOs) are expressed in a midrange theory that hypothesizes what contextual influences (C) have triggered the mechanisms (M) that result in certain outcomes (O). The hidden or implied nature of the mechanisms makes this the most challenging aspect of a realist analysis. Once an initial midrange theory of the connections between the C, M, and O is proposed, the theory is revisited and adjusted iteratively as refinements come to light through reviewing the literature or in the further analysis of data.

A realist approach to research and evaluation has been applied in the field of participatory action research in several ways. Realist syntheses have been conducted to explore the effectiveness of community-based participatory research approaches,^4,5 and a participatory approach has been used to conduct a realist review.⁶ Several studies have combined participatory research and realist approaches. Mutschler et al.⁷ refined an existing realist theory of a psychosocial rehabilitation program by examining mechanisms of change and recovery outcomes quantitatively. Members of the program participated in research design, recruitment, data collection and dissemination. Issues of having to balance rigor with the time and engagement demands of working collaboratively with an organization were raised. However, as is often the case for participatory research projects, organization members were not part of the initial data analysis step. Westhorp et al.⁸ conjoined realist principles with action research to test a program theory for a service innovation. They linked realist aspects to each part of the iterative cycle of action research (plan, act, observe, reflect, change plans) that engaged both researchers and program professionals and participants in a co-learning process.

The Westhorp team⁸ conducted workshops with stake-holders for each stage—a situational analysis, prioritizing the issues of concern, co-design, and trialing and refining ideas for change to the interventions. This model of participant engagement in every action research step was necessary for staff to implement service improvement and they successfully developed substantive theories that embodied the context– mechanism–outcome configurations of a realist approach. Westhorp’s team⁸ involved program staff in the development of theories and designs of the intervention but they did not indicate whether staff partners were involved in analyzing interviews using a realist approach. In this article, we present our experience of developing procedures that enabled participation of a community partner and individual community members (grassroots leaders) in a realist-oriented analysis of qualitative interviews.

There are challenges related to qualitative data analysis using a realist approach given the complex nature of distinguishing CMOs and their connections. These linked CMOs form the basis of developing hypotheses that underpin the design and implementation of an intervention or program. Despite these challenges, using a realist approach in community-based research is valuable because it can address the community members’ questions of what works for whom under what circumstances. Given the value of this approach and a commitment to community-engaged research, we wanted to develop more accessible analysis procedures.

Even with the best intentions, participatory research processes often reserve the analysis stage of research for the university-based members of the research team. However, there are ways to prepare data so they can be grouped and organized into insightful patterns by mixed groups of community members, professionals and researchers.⁹ Given the extra complications of using a realist approach during qualitative analysis when we want to include community partners, the purpose of this article is to describe how we adapted our analysis procedures to be both realist and participatory.

METHODS

A partnership between the Centre for Connected Communities (C3) and the Dalla Lana School of Public Health at the University of Toronto was in place since 2017 to explore community resilience in the face of climate change. C3 was a community development strategy organization whose purpose was to influence systems so that they increasingly refocused power and resources to put communities that have been historically marginalized, racialized and made vulnerable at the center. C3 did this by “connecting community builders with knowledge, research, tools and each other, translating knowledge across sectors, and celebrating and elevating the work of community building as some of the most important work in our society so that communities can find collective solutions to complex social issues.” ¹⁰ The C3/Dalla Lana School of Public Health partnership developed as an iterative process beginning in 2016. The leads in this partnership prioritized identifying a shared purpose and guiding principles for our work together across several projects, and engaged in intentional actions to foster reciprocal learning and trust building. The capacity of these two organizations to work together effectively was further strengthened in 2017 through 2019, when we worked on a literature review that resulted in the article: Citizens and Formal Institutions Working Together to Build Community-Centered Resilience.¹¹

In 2020, we were able to take advantage of this existing partnership to rapidly mount a qualitative research study of grassroots leaders across six neighborhoods in Toronto in the summer of 2020 in the context of the first wave of COVID-19. The six neighborhoods were selected because we knew there was an active grassroots response to the pandemic, they are considered marginalized within the City of Toronto, and each neighborhood has a unique history and different types of social infrastructure in place. The neighborhood leads were recruited from a city-wide network of grassroots leaders called Local Champions Network, which was supported by C3. The neighborhood leads in Local Champions Network tapped into their networks to recruit grassroots leaders from each community to be interviewed for this study, review and critique findings, and disseminate resulting information throughout their networks. The criteria for participation in the research were a) the grassroots leaders lived in one of the six identified communities and b) they were engaged in neighborhood pandemic responses. C3’s role included convening the neighborhood leads several times to ensure they had a solid grounding in the purpose of the research and the approach we were taking, were able to actively participate in analysis sense-making and felt included, supported, and heard as the project evolved. Neighborhood leads who recruited others were compensated for their time via a paid contract and those interviewed were paid an honorarium. The preliminary hypothesis that we explored was that the history of community organizing and prior relationships with formal city-level institutions affect the ability of grassroots leaders to be two-way connectors of pandemic information, community needs, and resources to reach those most in need.

A total of 55 people were invited and 46 grassroots leaders completed an interview on Zoom or by telephone. All interviews were conducted by G.M., a university-based researcher, who was paid as a project manager. Interviewees were asked questions about how the first wave of the COVID-19 pandemic affected their neighborhood, the actions they took, and what helped and hindered their actions. Consistent with a realist approach, the purpose of the interviews was to explore the contexts in these different neighborhoods, the mechanisms underpinning grassroots actions, and the outcomes they observed. The six neighborhoods had different kinds of community organizations, varied histories of resident engagement, as well as varied prepandemic relationships with formalized institutions, and we hypothesized that this would affect the nature and effectiveness of the grassroots leader actions. From four to twelve grassroots leaders/residents were interviewed in each of the six neighborhoods. Ethics approval (protocol #39393) was received by the University of Toronto Health Sciences Research Ethics Board.

Initial CIMO Hypothesis

Because each neighborhood has a different configuration of grassroots leaders and community organizations with different relationships to institutions and municipal government (context), the different relationship networks and histories of community organizing (mechanisms) affect the ability of grassroots leaders to be two-way connectors of pandemic information, community needs, and resources (intervention actions) to respond efficiently and effectively to local need during the COVID-19 pandemic (outcome).

Analysis

The preliminary hypothesis stated above that connected contexts, mechanisms, actions and outcomes, was well suited for a realist approach to the analysis.^2,3 Much of the literature describes methods for conducting realist reviews and syntheses which are useful for qualitative data analysis. We used these guidelines as a foundation for our realist approach to interview analysis and to capture the implicit or explicit connections made by interviewees when they described what they did in their communities, what worked and why or why not. The purpose was not for the research team to develop the CIMO connections but for the team to uncover the implicit or explicit connections made by interviewees. This is built on the premise that each grassroots leader that was interviewed has developed reasoning that underpins their work based on their experience and we wanted to explore how similar these underlying theories were and how much they differed based on contextual differences between the six neighborhoods.

To assist practitioners and community partners make distinctions and links between mechanisms, actions and outcomes, we added the intervention features (I) into the CMO configurations to create CIMOs as described by Punton et al.³ Initially, each sentence was coded for context (C), mechanism/intervention (M/I), and outcome (O) components with sub-codes to identify unique contexts, mechanisms/intervention components and outcomes (e.g., C1, M2, O13) as per the method used by Jackson and Kolla.¹² The various connections made by each interviewee were complicated and generated too many different codes in the first pass through a couple of interviews and the process was stopped. There were several reasons why this approach was abandoned early in the analysis: a) we were on a timeline to report back quickly to the grassroots leaders, given the need for information relevant to the pandemic, b) C3 felt the level of detail (too much detail) did not meet their needs for testing the preliminary hypothesis and reporting back to the neighborhoods involved, and c) it was challenging for everyone on the team to understand and agree on the type of coding required. At this point, we agreed to have only the first author do all of the initial coding.

Rather than create a detailed codebook with a tree of sub-codes, we opted to identify contexts (C), mechanisms/interventions (actions) (M/I), and outcomes (O) and their connections for each paragraph in each interview as described by the interviewees and to put them in a comment box using Microsoft Word. In a separate document, a paragraph was written that summarized all of the contexts mentioned by that interviewee; a list of the actions and interventions taken and potential mechanisms that supported those interventions was created; and the outcomes observed were itemized in point form for each interview. Based on this material, from one to seven midrange context–intervention–mechanism–outcome (CIMO) hypotheses were created specific to that interviewee by the first author. In some cases, the interviewee may have explicitly made these CIMO connections, and in other cases, the connections were inferred.

The initial data analysis summaries with CIMO hypotheses were conducted by the lead author. Subsequent review of these summaries and hypotheses was conducted by a core research team made up of three academics with community development, public health, resilience and human geography expertise and three representatives from C3 whose expertise focused on the ways in which community social infrastructure enables positive community building by grassroots leaders and groups.

Each member of the core research team took one neighborhood summary to review and identified key insights across the interviews from that neighborhood. No tools or guiding documents were used. The team met to review these insights, integrate both academic and on the ground perspectives on the themes that emerged, and discussed the similarities and differences across all six neighborhoods. These insights served to refine the initial hypothesis and to identify the mechanisms underpinning the success or lack of success of the community responses to the pandemic. Several iterations of ways to organize the information into themes were required before the team was satisfied with the result.

The team’s insights served as an organizing framework for grouping the CIMO hypotheses into themes. These themes were presented and discussed with one or two grassroots leaders drawn from each neighborhood, approximately a quarter of the total interviewee pool, in an evening session of sense-making over Zoom. The sense-making session confirmed the direction the research team had taken and the interviewees mainly repeated the same messages they had given in their interviews. From a participatory research perspective, the session could have been more interactive, if we had been able to work with pieces of data as a group in person with facilitated discussion. We did break into three small groups over Zoom with each facilitated by a university researcher but the time was short and there were technical issues. After the interviewee consultation, these key insights were further revised and refined over several meetings by the research team.

This research generated several midrange theories about: (a) the relationships of grassroots leaders, community organizations, service providers and city institutions (briefly illustrated in this article); (b) the nature of community resilience and the connected community approach; and (c) the role of racialization and long-standing structural inequities affecting the six neighborhoods studied.

One of the Refined CIMO Hypotheses after Participatory Analysis Process

In a pandemic presenting a completely new situation (context), community organizations with preexisting positive relationships with service providers, and excellent relationships with local residents (mechanisms) were able to communicate across the community, coordinate volunteers and grassroots leaders to gather information about resident needs, use their structures to bring in special funding, work collaboratively with various groups and organizations, and enable residents to participate in local planning (actions) toward making sure that there were two-way communications about pandemic information and community needs, support for grassroots leaders, and distribution of resources to those who needed them the most (outcomes).

DISCUSSION OF REALIST ADAPTATIONS FOR A PARTICIPATORY RESEARCH PROCESS

There are several places where special adaptations were made to make this realist analysis a participatory process. As is typical for a participatory research process, the community partner, C3, was involved from the beginning of this project as a lead in the overall conceptualization of the research, its design, and the data collection phase. Table 1 lists the typical steps in qualitative analysis and compares the typical process used in participatory research, realist research, and the participatory realist approach used in this project.

Each approach listed in Table 1 has its advantages. The main contributions of our participatory realist process is that a) we used emergent coding in Word, rather than a software program such as NVivo as a way to make the CIMO coding more accessible, and b) we gathered insights from community research team members and a proportion of the interviewees to further refine the resulting hypotheses.

There are three key areas of original contribution from our project. The first is that, in general, academic researchers are seen to have the expertise to interpret and code interview data either in typical qualitative analysis or using a realist approach. In particular, given the complexity of understanding the nuances of contexts, mechanisms and outcomes in the realist approach, this expertise is paramount. In the case of realist synthesis of the literature, much of the literature is missing explicit descriptions of contexts and mechanisms and the analyst must infer this information.² In the detailed guidelines for conducting a realist evaluation, Wong et al.² talk about testing their initial hypothesis by examining the data iteratively, going back and forth between inductive and deductive reasoning. In our process, the analysis was designed to identify the natural way that interviewees talked about connections between the factors in their context, the people they worked with, the actions they took, the results they observed and their underlying reasoning about why this worked (or not).¹² The exposure of this tacit knowledge in the form of hypotheses was understandable and useful to community members.¹³

Second, this summary approach to analysis took far less time than the detailed realist coding method published by Jackson and Kolla.¹² The latter took about 1,000 hours part-time over a period of 3 years. The approach reported in this article took approximately 200 hours part-time over 5 months. It is not only useful in terms of engaging community members, but it is also more efficient.

Third, we designed procedures that made the realist approach understandable and did not require every member of the team to have expertise in realist or qualitative analysis. By creating summaries of the contexts, mechanisms/actions and outcomes for each interview as well as several midrange theories that connected the contexts, mechanisms/actions and outcomes from the perspective of each interviewee, all other members of the research team could review this and look for common elements and differences. The analysis information was in a format that was easy to grasp and work with by the C3 team members and a selection of the interviewees. A limitation of this work is that during the first stage of analyzing interviews, someone with expertise in using the realist approach is required.

Table 1.

Comparison of the Procedures in Qualitative Interview Analysis that are Participatory, Realist and Combined Participatory/Realist

Fourth, we shifted our approach from one of academics “allowing or facilitating community participation in their process” to one of “centering the community view and input in steering the process and deciding when it is appropriate to be involved.” From the beginning, both partners collaborated to develop the research question and the overall design, including the analysis. In the analysis phase, there are several steps—(a) data sorting, (b) arranging into clusters/themes, (c) presentation and explanation of the big picture with a story/narrative. C3 was involved in step (c) and trusted the academics to do the first two steps. C3 states that “being involved at all steps of the process from the design of the research questions to the analysis to the writing meant that we, as a community partner, felt fully part of the research team.” C3 learned a lot from the analysis process which allowed us to fully take into consideration the nuances of context, mechanism, intervention and outcome when contributing to the writing of the various articles we have published out of this work.

The realist method advanced C3’s work by engaging community members as experts about their own context, reflecting the reality that there are multiple ways of knowing; simultaneously offering academic rigor while at the same time learning from and crediting the wisdom generated by lived experience. We also believe that the approach we used modified the results of a typical realist evaluation approach. An academic perspective on inferred connections between contexts, mechanisms, interventions and outcomes could set up hypotheses that do not recognize some of the subtleties of the linkages. By including community members in the overall research team and in a sense-making workshop, we were able to hear the connections reiterated and adjust the presentation and narrative accompanying the results (largely in the form of hypotheses).

One of the most valuable aspects of realist evaluation is that it provides a rigorous alternative to randomized control trials and other esteemed quantitative methods for determining the value of community engaged work.² The realist approach recognizes the role of theory and hypotheses in determining the impacts of community work and that the outcomes for the same types of community processes differ because the context for each community differs. Our work unpacks some of the mystery of how this works so that there can be more engagement of community partners in a participatory research process.

CONCLUSIONS

A realist approach can be useful for community organizations who want to explore what works for whom and under what circumstances in their communities. The key challenge is the complexity of the realist approach to data analysis and the necessity to engage academic partners to use this approach. We believe that our procedures enabled the realist approach to be understandable and useful as part of participatory action research, even in the analysis phase. This opens up the possibilities for participatory researchers to explore this fruitful approach to better understand community processes.