Web-based Survey Data Collection With Peer Support and Advocacy Organizations:Implications of Participatory Methods
Background: The 2012 National Survey of Peer-Run Organizations is one of the first to survey a nationally representative sample of mental health peer-run organizations, nonprofit venues for support and advocacy which are defined by people with psychiatric histories being in positions of authority and control.
Objectives: This paper describes data collection methods and demonstrates how participatory strategies to involve people with psychiatric histories intersected with Internet research to achieve study aims.
Methods: People with psychiatric histories were involved in designing and implementing a web-based survey to collect data on peer-run organizations' operations and views on national policy. Participatory approaches were used throughout design, data collection analysis, and dissemination.
Conclusions: The extensive involvement of people with psychiatric histories in project design and implementation were important strategies that contributed to this study's success.
Mental health services, health care economics and organizations, health care surveys, community-based participatory research, organizations
Mental health peer-run organizations are nonprofit venues for support and advocacy,1 defined by people with psychiatric histories in management, oversight, and staffing.2 The 2012 National Survey of Peer-Run Organizations is one of the first to survey a nationally representative sample of such organizations.3 The aims of the study were to document organizational operations and managements' attitudes toward policy change. Findings related to study aims and substantive analyses are available in other manuscripts.2-4 This was the first such survey since the 2002 Survey of Organized Consumer Self-Help Entities conducted by the Substance Abuse and Mental Health Services Administration (SAMHSA).5 Other studies of similar organizations focused more locally and had relatively small sample sizes, which did not capture nationally representative perspectives.6-8
The 2012 National Survey engaged representatives of peer-run organizations to provide data that could be used to inform decisions about federal and state policy priorities and to document the operations of these organizations as the healthcare system prepared for implementation of the Affordable Care Act (ACA). This paper describes how the 2012 National Survey incorporated participatory research strategies involving people with psychiatric histories and how this intersected with Internet research, resulting in a high-quality research study inclusive of community members. The aim of this paper is to walk the reader through each step of the research process, how community members were involved, and what the contributions of participatory research were to the project.
The target for this study was peer-run organizations in the United States in 2012. For the purposes of this research, [End Page 45]
an incorporated, independent nonprofit organization or a non-incorporated organization that operates independently from a non–peer-run parent organization; at least 51% of the board of directors or advisory board are people with a psychiatric history; the director is a person with a psychiatric history; and most staff members or volunteers have a psychiatric history.3
PARTICIPATORY RESEARCH, USER-LED RESEARCH
Participatory research uses perspectives and involvement from community members to acquire culturally relevant and actionable data. This study involved people with psychiatric histories in every stage of the project, consistent with guidelines on participatory disabilities research.10 The involvement of people with psychiatric histories in organizational management and direct support is a core element of peer-run organizations; similarly, the involvement of people with psychiatric histories in participatory research is imperative, making this research approach a good fit to study these organizations.11 Peer support initiatives has been a particular emphasis of user-led research, which attempts to shift control, rights, and knowledge to service users.12
"User-led research" refers to projects using professional researchers who are people with disabilities as project leaders or co-leaders13; it is a type of participatory research. In contrast with other participatory approaches that vary in the amount of user involvement and control,14 user-led research incorporates individuals with professional research qualifications and life experiences of the community under study.15 Issues that interfere with user-led research include discrimination in higher education and employment, stigmatizing views of people with psychiatric histories as "biased" observers, and skepticism of the scientific method and its implications by people with psychiatric histories.16 The 2012 National Survey was carried out in an academic setting, with the buy-in and participation of people with psychiatric histories.
CHALLENGES IN RESEARCH ON PEER SUPPORT ENDEMIC TO THE CONSUMER/SURVIVOR MOVEMENT
The terms "consumer" and "survivor" refer to people who have experienced emotional and psychological distress that led to being labeled with mental disorders (often called "serious mental illnesses").17 "Consumer" and "survivor" are terms used in a social change movement called the "consumer/survivor movement."18
Peer-run organizations grew out of decades of consumer/survivor advocacy for nonmedical alternatives to the psychiatric system,18,19 and these organizations' goals align with the movement's emphasis on self-determination and community leadership.20-22 The objectives of this study were highly relevant to the consumer/survivor movement because peer support is a high priority. Peer support engages people in mutually supportive nonmedical relationships based on respect, shared responsibility, and agreement of what is helpful.23
There are challenges to recruitment and questionnaire design for these organizations. They tend to operate informally and take diverse approaches, which can make them difficult to identify as distinct entities providing a discrete service.2 The nonhierarchical relationships within peer-run organizations18,24-26 that grew from responses to experiences in the psychiatric system20 can make them difficult to identify and study in large-scale research studies.3 The 2012 National Survey addressed these challenges by marrying web-based survey research with participatory research. Web-based survey research provides adaptable question design and makes it affordable to recruit large numbers across a large geographic area like the United States. This makes web-based surveys a particularly good fit for a study such as this.27
This paper reviews how every step of the study—motivation, design, data collection, data analysis, and dissemination—used modern survey methods that included members of the target population in designing and conducting the research.
Participatory Design Components
The study's design, data collection, and interpretation were informed directly by policy, advocacy, and administration stakeholders, most of whom identify as people with psychiatric histories. We based our inclusive approaches on an existing framework for participatory research with people with disabilities.10 Stakeholders who contributed to project development were identified through the lead researcher's professional network, based on their history of contributions [End Page 46] to the field, rather than a singular "partner agency" with the university hosting the research project. Table 1 outlines the participatory design components10—as applied to study procedures—used in the survey.
In this section, we describe the research process and how participatory strategies were used in design, data collection and analysis, and dissemination. Figure 1 displays the steps in designing the study and collecting data.
This research study was determined "Not Human Subjects Research" by the Institutional Review Board at the Johns Hopkins School of Public Health, as defined by the Department of Health and Human Services regulations 45 CFR 46.102, and did not require institutional review board review.
Researcher Translation From Participating in Community Networks
Researcher translation refers processes in which researchers join community members in discussing important issues and potential research topics.10
Developing Specific Aims. The lead researcher previously worked on smaller survey research projects with members of the target population and was aware of issues that they might like addressed in a large-scale survey. Discussions within this community—via listservs, state/local organizing meetings, and national conferences—was the primary motivation for questions about the ACA and other policy priorities, and was consistent movement advocates' writing since the 1970s.20 Other questions were related to organizational operations, such as staffing, financing, and support provided, because there has been no documentation of these characteristics of peer-run organizations since 2002.5
Inventory of Organizations. The project capitalized on existing relationships in the national network of consumer/survivor advocates and peer-run organizations to create an inventory of all peer-run organizations in the country to conduct recruitment. At the outset, there was no comprehensive national list of peer-run organizations. Our search identified 895 organizations and programs. The primary strategy was [End Page 47] contacting statewide consumer networks and state offices of consumer affairs from August 2010 to June 2012. The National Coalition for Mental Health Recovery (NCMHR) and officials with SAMHSA's Statewide Consumer Networking grant program and the Center for Mental Health Services' Office of Consumer Affairs were consulted. Statewide networks were contacted by email and phone to obtain lists of peer-run organizations in each state. This search method was supplemented with lists from SAMHSA's three National Consumer Technical Assistance (TA) Centers. In states without a statewide network identified by SAMHSA or NCMHR, key local informants and organizations were contacted to provide missing information.
An agenda-setting survey approach involves reviewing literature and conducting interviews or focus groups with members of the target population to develop a community-oriented research agenda.10
Instrument Design. The survey was designed to collect current information on organizational operations, including funding, staffing, services, and activities. An additional goal was to assess attitudes toward upcoming changes associated with the ACA that may affect peer-run organizations, to make policy recommendations to advocates and policymakers.
Individuals working in research, government, managed care, advocacy, and program development contributed to the design of the instrument. Almost one-half of those (18 of 37) were people with psychiatric histories and professional qualifications. These key informants were interviewed by phone or in person in an unstructured manner. They were asked to identify topics and questions most pertinent to this population in the current policy context. Some sections of the survey were adapted from SAMHSA's 2002 Survey of Self-Help Organizations. Sections asking about consumer involvement in management and decision making were adapted for a web-based survey from the Consumer-Operated Services Program multisite evaluation fidelity instrument.28
The lead researcher designed the initial version of the instrument, [End Page 48] which underwent expert review, pretesting, and field testing before being launched online via Qualtrics survey software.
People With Psychiatric Histories as Research Team Members and Consultants
Researchers who identify as people with psychiatric histories had significant control over the research process and allocation of resources, and were paid team members and consultants. Experts identifying as people with psychiatric histories served as consultants on research process and content to improve social validity. These consultants had content experience, familiarity with interrelations among issues, and a nuanced understanding of the study content and implications for research.
Project Lead. The lead researcher identified as a person with a psychiatric history. She made determinations—in consultation with other researchers and team members—about funding allocations, hiring team members, data collection methods, and interpretation and dissemination.
Research Team. Research assistants were identified by On Our Own of Maryland, Inc. (a statewide consumer network) and hired by the lead researcher to assist in data collection. The research assistants identified as people with psychiatric histories and had experience in peer-run programs or as advocates. They were trained in a protocol for follow-up with nonresponders that included timing of follow-up contacts and a script.
Recruitment and Data Collection. Data collection was conducted from April to October 2012. The survey could be completed anywhere with an Internet connection, and respondents could pause during the survey and come back later with their answers saved. Funding was allocated by the lead researcher to pay research assistants for follow-up with nonresponders, provide an incentive for participation, and conduct public dissemination.
Organizations were recruited through a letter on university letterhead sent via the U.S. Postal Service introducing the study, its methods, and purpose, and the fact that it was being conducted by researchers with psychiatric histories. The letter, which included a $5 cash incentive, stated that participants would receive an email with a link to the survey, to be completed by the executive director or designee. The invitational letter was endorsed by the SAMHSA consumer TA centers and the NCMHR. Follow-up with nonresponders included multiple e-mail and phone contacts by research assistants.
Because research assistants were familiar with peer-run organizations, they were sensitive to the concerns and questions of the organizations' directors. For instance, research assistants could understand and respond nondefensively to concerns about the history of abuse in research on mental disorders or the cooptation of self-help initiatives by traditional medical care providers.
This study achieved a final response rate of 80%; 380 peer-run organizations were in the final sample for analysis. Response rates for web-based surveys of organizations range from 68% to 89%, and response rates for surveys of individuals average about 40%.27,29
Consensus Panel. We convened a consensus panel of five advocates, TA providers, program directors, and researchers with psychiatric histories who are well-known advocates for fidelity to peer-run initiatives and meaningful inclusion of consumer/survivor voices in policy development. The consensus panel decided on final inclusion criteria, unanimously endorsed by all members. The panel confirmed the definition of a peer-run organization was confirmed by these community members and was based on existing user-led research on fidelity to organizational structure.28 The consensus panel based their discussion on aggregate data provided by the research team on program structures and policies, and on the consumer/survivor movement's history of creating independent alternatives to traditional mental health services. This improved the validity and applicability of the definition, because it emerged from members of the community.
Qualification for Study Participation and Data Preparation. Organizations and programs that did not meet peer-run/peer-operated criteria were excluded from the sample. All organizations suggested by statewide consumer networks or other informant were recruited. However, we could not verify that all participants qualified for the study, so we empirically determined inclusion in analyses using the consensus panel's criteria.
Product Champions for Buy-in and Dissemination
Product champions are individuals or groups representing the target population who find the research results useful to them and their causes and, therefore, support the research process and dissemination of results.
Recruitment Buy-in. The co-signing of the invitational letter by the SAMHSA consumer TA centers and NCMHR [End Page 49] was intended to gain the trust of respondents and signal the importance of the study. However, not all respondents necessarily identified with these entities.
Dissemination. In addition to articles in academic peer-reviewed journals,2-4 public reports on organizational characteristics and attitudes toward the ACA were created and disseminated immediately after data were analyzed. Participants were directly emailed reports, also available on the web, with a limited number of free paper copies available.
Immediately after completion of data collection and initial calculation of descriptive statistics, the lead researcher presented these preliminary results at the Alternatives 2012 conference, a SAMHSA-sponsored annual national conference of people with psychiatric histories, many of whom work in the mental health system or peer-run organizations. Attendees provided valuable input on interpreting results from their perspective as survey respondents, which was incorporated into written reports of study results.
Results were presented at the American Public Health Association 2013 annual conference by the lead researcher, a participant, and a member of the consensus panel, and by the lead researcher at the American Public Health Association 2014 annual meeting. The Association for Behavioral Health and Wellness (a membership organization of behavioral health managed care companies) hosted a webinar for members to learn about results related to the ACA and how they might implement recommendations.
Lessons for User-led and Participatory Research
The involvement of people with psychiatric histories in project design and implementation highlights important strategies that contributed to this study's success. Although the expertise of both researchers and community members is imperative to participatory research, the inclusion of individuals from the community as professional researchers may make user-led research a more advanced form of participatory research in mental health.12,13,15,16,30,31 Although it is not always possible to include community members in projects in substantial numbers, owing to availability of research expertise, appropriate partners, financial resources, and other facors,32 this project demonstrates additional benefits of user-led participatory research, including engagement of an array of advocacy and service organization representatives in research design and dissemination, and hiring and supervision of research assistants from the target population.
The study's specific aims for data collection and analysis were motivated by the passage of the ACA and ensuing discussions among peer-run organizations and advocates about potential challenges related to Medicaid reimbursement. Although not commonly thought of as a service provider type that would be impacted by insurance reimbursement changes, discussions among peer-run organizations inspired the need for a national study on these issues. This approach may be useful in other policy research addressing regulatory and financing changes affecting peer-run organizations or peer support to solicit ideas from the impacted community, where the issues are well-understood by the researchers before embarking on the project.
Existing relationships in the community and project staff who are part of the target population were key to achieving a high response rate. Team members' existing relationships helped to identify organizations and encouraged survey responses. Other research suggests that a shared history of marginalization is relevant to data collection, enabling better connection with respondents13,33 who may be more likely to open up to interviewers that are their peers.34 This reflects our experience.
The consensus panel was convened to determine inclusion criteria. A consensus panel may be useful when a study needs a definition of terms or constructs that is relevant to a community. The consumer/survivor movement is a close-knit community; members often have long histories of collaboration. This can make consensus easier (i.e., owing to commonly held beliefs) or more challenging (i.e., owing to political fragmentation common to social movements).35 It might have been useful to have convened an advisory group for more substantial, formal roles in an ongoing manner throughout the project. For instance, although many knowledgeable individuals gave input into the survey design, members of the consensus panel decided that the values of peer-run organizations were not measured adequately. In the future, such a group would be helpful in brainstorming questions on "values," while limiting social desirability bias.36 [End Page 50]
Engagement and Dissemination on a Broader Scale
This project did not use the common structure of a participatory research project that engages with a limited number of local or national community partners. Instead, we capitalized on the longstanding professional relationships of team members with established professionals in the consumer/survivor movement. This strategy also supported dissemination to national conferences and groups that could implement policy recommendations.
This study successfully involved people with psychiatric histories in collecting national data on peer-run organizations from a web-based survey. The availability of affordable and easy-to-use web survey tools is a good fit for participatory national research with geopolitically diverse social service organizations. The strategies described herein contributed sensitivity and insight needed to design an appropriate instrument, inventory potential respondents, and implement successful recruitment techniques.
ACA-Related Advocacy in the States
An essential purpose of this study was to provide nationally representative data to advocates and policymakers to inform perspectives and make data-driven arguments in public policy deliberations about the future of peer-run organizations. Since the study's conclusion, there has been rapidly increasing interest in the states in Medicaid-reimbursable peer support. Although the focus has mainly been on peer support within traditional mental health services, peer support provided by peer-run organizations will also be affected by state laws. The survey results provide a broader perspective on the pros and cons of this issue, rather than just soliciting input from a small group who have the opportunity to influence policy.
Usefulness of Enumeration and Updating Database/Survey
Updating the database of peer-run organizations is valuable; it is important that these groups continue to be enumerated and monitored for changes in organization and practice associated with changing policies. However, there is currently no known effort to update or create ongoing national databases of these organizations.
Since completing the project, many statewide networks have expressed interest in receiving updated lists to use in organizing and coalition building. An updated database would be essential for a follow-up wave of the study. Even with buy-in from the community, the enumeration process is an intensive research process in itself.
The study found that web-based surveys are feasible in organizational research, even with hard-to-reach populations, provided researchers use complementary outreach and inclusion approaches. We believe that multiple contacts by research assistants, particularly repeated phone calls, resulted in the high response rate.
The study demonstrates the feasibility of monitoring a group of peer-run organizations using a web-based survey and facilitating nationwide assessment of organizational operations and attitudes toward policy changes. The 2012 National Survey of Peer-Run Organizations addressed some inherent challenges by marrying web-based survey research with participatory research. Web-based survey research provides adaptable question design and makes it affordable to recruit large numbers across a wide geographic area, making them a particularly good fit for studies like this.27 We believe these factors contributed to the enthusiasm for the survey evidenced by the high response rate. Future research could test the effectiveness of user-led, participatory, and traditional research approaches more thoroughly to distill important elements.
This study was funded by grant T32MH019545 from the National Institute of Mental Health and a grant from the Johns Hopkins Center for Mental Health Initiatives. The authors thank members of the consensus panel: Dan Fisher, Lauren Spiro, Darby Penney, Sera Davidow, and Christa Burkett. The authors also thank research assistants John Otenasek, Cheryl Gottlieb, Bob Stockfield, and Cathy Levin.