Formative Evaluation of a Grocery Store-Based Nutrition Incentive Program in a Primarily Rural State

Summary

Increasing the reach of nutrition incentive programs (NIP) requires their successful implementation beyond farmers markets into grocery stores. This report describes our approach to formative evaluation and key lessons during the expansion of the Double Up Oklahoma (DUO) NIP into 10 grocery stores over a two-year period.

Background

Nutrition incentive programs (NIP) are designed to support nutrition security among Supplemental Nutrition Assistance Program (SNAP) participants by reducing financial barriers to purchasing fruits and vegetables (F/Vs).¹ Initially debuted in 2009 at farmers markets through Double Up Food Bucks² and similar initiatives, NIP have successfully scaled to operate at an estimated 2,150 farmers markets nationally as of September 2023.³ However, most SNAP participants prefer to shop at grocery stores over farmers markets for multiple reasons, such as longer operating hours, social acceptability, and perceived differences in food safety, quality, cost and variety.⁴ Thus, to broaden the reach of NIP within the SNAP population, the Gus Schumacher Nutrition Incentive Program (GusNIP) enabled NIP expansion to brick-and-mortar food retailers in 2019, which now include over 1,500 participating food retailers across the U.S.³

Although all NIPs incentivize F/V purchases using a similar structure (e.g., spend $1, get $1), best practices for successful NIP implementation at farmers markets cannot necessarily be translated to brick-and-mortar locations. For example, farmers markets use pre-paid token or scrip systems that issue same-day nutrition incentives that can be immediately redeemed while shoppers visit booths,⁴ but grocery stores can distribute incentives via loyalty cards or vouchers at the point-of-sale (POS) for redemption during a separate transaction or at a later shopping visit. Best practices for the promotion of these programs to eligible consumers may also differ, since farmers market patrons represent a niche segment who may favor and eat F/V more than those who do not shop at farmers markets.^4,5 Finally, the type and quality of F/V can differ between farmers markets and grocery stores, which may influence NIP participation rates.⁴ All of these factors, including the place, people, and product, may affect whether and how desired program outcomes are achieved.⁶ Thus, formative evaluation activities, including details about how the program was implemented according to plan (process evaluation)⁷ and identifying opportunities to respond flexibly to overcome implementation barriers (developmental evaluation),⁸ can be useful during the early phases of NIP implementation in grocery stores as novel settings.

Oklahoma, a mostly rural state that ranks 49^th in F/V intake,⁹ operates one of the largest NIP nationally, Double Up Oklahoma (DUO), which expanded from farmers markets to grocery stores beginning in 2020 as a result of receiving a $500,000 GusNIP grant. Further expansion was made possible though a $650,000 GusCRR (GusNIP COVID Relief and Response) grant in 2021, and a GusNIP grant of $14.2 million in 2022, the largest award of its kind at the time.¹⁰ In this Report from the Field, we describe our approach to understanding the context and challenges of NIP expansion into grocery stores through formative evaluation, including process and developmental evaluation activities performed during the first two years of DUO expansion into these settings. We conclude with key lessons that resulted from these evaluation activities.

Double Up Oklahoma (DUO) Grocery Store Expansion

Community partnerships

Hunger Free Oklahoma (HFO) is a statewide, non-profit organization whose vision is the end of hunger in Oklahoma.¹¹ In 2019, HFO began administering DUO at select farmers markets with a goal for later expansion into grocery stores. Homeland Acquisition Corporation (HAC), which operates over 60 grocery stores in Oklahoma, including many located in United States Department of Agriculture-designated low-income, low-access areas,¹² agreed to work with HFO to prepare and develop a point-of-sale (POS) system capable of incentive issuance and meeting redemption tracking requirements. Through the POS system, SNAP purchases made at participating HAC stores are matched dollar-for-dollar (up to $20 per day) with physical vouchers that can be used for subsequent F/V purchases within 60 days.

Implementation timeline

In November 2020, DUO was beta-tested in a pilot grocery store that resulted in multiple refinements to the POS system, reporting, training, and marketing over a four-month period. Double Up Oklahoma was then launched through a staged roll out into nine additional stores by the end of 2021 (program year one; PY01). The 10 stores (Table 1) included two supermarket store brands (n=6 stores),

Table 1.

IMPLEMENTATION DETAILS AND COMMUNITY CHARACTERISTICS FOR 10 GROCERY STORES PARTICIPATING IN THE DOUBLE UP OKLAHOMA (DUO) NUTRITION INCENTIVE PROGRAM ACROSS OKLAHOMA.

Figure 1.

Example promotional "Iron Man" poster to promote the Double Up Oklahoma (DUO) nutrition incentive program in grocery stores.

a discount grocery store brand (n=3), and a community-based, small-format store operated by a local non-profit organization (n=1). One discount store closed after six months of DUO and no new stores were added during program year two (January 2022–December 2022; PY02). Among stores selected for DUO implementation, the county-level SNAP participation rate ranged between 9.2% and 21.1%¹³ and the census tract-level poverty rate ranged between 13.3% and 52.8%.¹⁴

Prior to the launch of the program at each store location, grocery store personnel received an orientation to the program by HAC management. Cashiers received a separate, DUO-specific training and were expected to inform all SNAP customers about the DUO program during checkout. With input from health promotion scientists, community advisors, and behavioral nudge experts, and in collaboration with HAC marketing leadership, HFO developed and provided a variety of promotional posters for display throughout the stores. These included two sizes of posters (11ʺ×17ʺ and 22ʺ×28ʺ "Iron Man") (Figure 1) for high-traffic locations near entrances, cash registers, and produce departments. Iron Man posters were two-sided and included DUO program policies on one side. To help reach those SNAP customers who do not visit the produce department, 3ʺ×5ʺ behavioral nudge cards (25 varieties) were developed

Figure 2.

Example behavioral nudge cards and intended areas of placement to promote the Double Up Oklahoma (DUO) nutrition incentive program in grocery stores.

for display in non-produce departments and aisles throughout the store, such as the deli and cereal aisle, promoting the addition of fresh F/V to food and meals (Figure 2).

Formative Evaluation Approach

Process and developmental evaluation activities

The primary intended users of the formative evaluation include the program administrator, HFO, and the participating program retailer, HAC. Following input from these intended users to identify key questions, evaluation activities were led by an external team with extensive experience evaluating healthy-eating interventions across various retail settings. Key goals involved evaluating the process of program implementation as intended, including DUO program fidelity across three domains: 1) quality and availability of produce, 2) display of promotional signage throughout key areas of the store, and 3) promotion of DUO by cashiers at checkout. Additionally, a developmental evaluation goal sought to identify unanticipated events and challenges that might affect DUO outcomes.

To achieve these goals, trained graduate assistant evaluators conducted site visits bi-weekly on different days and times of the week, including weekends. Site-visit activities included an observational checklist and semi-structured, key informant interviews with people working in different grocery store roles (cashiers, produce manager, assistant manager, and store manager) to complement observational data. We routinely reviewed evaluation instruments to ensure relevant information was captured as the project progressed. All site-visit data were systematically collected and managed in a REDCap electronic database hosted by the University of Oklahoma Health Sciences.¹⁵

Process evaluation domains

The three primary domains are described in detail below.

Domain 1: Produce quality and availability

Evaluators rated overall produce quality as either "acceptable" or "unacceptable" at each site visit (Figure 3a) using an adapted approach informed by the Nutrition Environment Measurement Survey—Stores.¹⁶ We assessed produce availability on a three-point scale: "well stocked" (no stocking issues), "mostly stocked" (one or two produce varieties in need of restocking), and "needs restocking" (three or more produce varieties in need of restocking). Additionally, we recorded the number of pre-cut and pre-washed "value-added" options (Figure 3b).

Domain 2: Program signage

To evaluate the passive components of program promotion, evaluators documented whether a poster was "displayed," "not displayed," or "displayed with quality issues" for each desired location throughout the store. Similarly, evaluators documented whether behavioral nudge cards were posted in designated non-produce areas.

Domain 3: Cashier observations

To evaluate the active, interpersonal component of program promotion, site evaluators observed cashier-customer interactions during transactions at the POS. When multiple cashiers were present, all were observed at least twice, and at least four total observations were made at each site visit. Evaluators noted whether each interaction included five components listed in the cashier instructions in the DUO training materials (Figure 4); one additional component was added mid-PY02.

Key informant interviews

Key informant interview guides were added to the evaluation in mid-PY01 (July 2021) to help augment observational data. At each site visit, evaluators conducted up to four brief interviews with available store personnel who were involved with on-site DUO operations. Interview participation was not incentivized, but grocery store personnel were allowed to participate while on the clock. We asked position-specific questions for cashiers (6 questions), produce managers (6 questions), assistant managers (14 questions), and store managers (12 questions) (Box 1). Cashier interviews elicited insights regarding their experiences with the DUO program, including DUO-related POS system challenges, perceived barriers for customers, and successes of the DUO program. Although assistant manager and store manager interview guides contained several similar questions, those specific to assistant managers focused more on customer experience, whereas those specific to store managers focused more on staffing and training considerations. Produce manager interviews sought perceptions of DUO's impact on operations and inventory management.

Analysis

Formative evaluation data were analyzed in tandem with sales data as part of the program's overall evaluation strategy, in which we debriefed implementation

Figure 3.

Examples of process evaluation components related to product availability and quality collected during site visits to grocery stores participating in the Double Up Oklahoma (DUO) nutrition incentive program, including items from the observation forms used to evaluate produce quality (top) and example photo of a well-stocked value-added section of a produce department (bottom).

Figure 4.

Excerpt from site visit process evaluation instrument for documenting whether cashiers demonstrated expected promotional activities that were intended to raise Supplemental Nutrition Assistance Program (SNAP) customer awareness of the Double Up Oklahoma (DUO) nutrition incentive program at the point of sale.

partners weekly to provide timely feedback regarding on-site program operation and unforeseen developments.⁸ For the analyses presented here, we used STATA 16.1¹⁷ to calculate descriptive statistics for produce quality and availability, program signage, and cashier-customer interactions observed across all stores. We used QDA Miner Lite 6.0¹⁸ to code, sort, and group qualitative data from key informant interviews into emergent themes. These analyses focused on perceptions and barriers to program implementation and operation (developmental evaluation), as well as potential opportunities to improve the reach and effectiveness of the program (process evaluation).

Findings

Evaluators conducted 237 site visits across 10 DUO-participating stores during PY01–PY02, including 491 cashier observations (SNAP transactions) and a total of 650

Box 1. EXAMPLE QUESTIONS ASKED DURING KEY INFORMANT INTERVIEWS OF GROCERY STORE PERSONNEL WORKING IN DIFFERENT EMPLOYEE ROLES COLLECTED DURING FORMATIVE EVALUATION SITE VISITS FOR THE DOUBLE UP OKLAHOMA (DUO) PROGRAM

Table 2.

AGGREGATED QUALITY AND AVAILABILITY RATINGS OF PRODUCE AMONG ALL DOUBLE-UP OKLAHOMA (DUO) PARTICIPATING STORES, BY PROGRAM YEAR (TOTAL N=236 SITE VISITS).

interviews with cashiers (n=284), produce managers (n=152), store managers (n=119) and assistant managers (n=95).

Produce quality and availability

Produce quality was rated as acceptable on nearly every site visit (100% in PY01 and 98.3% in PY02) (Table 2) with unacceptable quality being limited to a few instances of overripe bananas and wilted bell peppers. However, evaluators noted produce needed restocking at one-third of visits during PY01, which improved to one-quarter of visits during PY02. While previously faced with challenges regarding produce shrinkage and waste prior to DUO, produce managers described new, positive challenges during interviews about keeping up with produce demands, particularly for value-added options, despite the hiring of new staff to assist with these products specifically (Box 2). Although site visits occurred during the COVID-19 pandemic, interviews did not attribute limited produce supply to distribution-chain issues, but rather store-level implementation barriers as they worked to fine-tune ordering and staffing to meet improved sales performance.

Produce managers were effusive with positive feedback regarding DUO's impact on their departments. Beyond notable improvements in sales figures, produce managers reported the ability to diversify offerings with more premium and novel produce options, such as pink pineapples (Box 2), as well as F/V options common to Hispanic (nopales, epazote) and Asian (bok choy, taro root) cultural foodways.

Promotional components

The two promotional components included observations of DUO signage and interpersonal interactions between cashiers and customers, in addition to interviews with grocery store employees.

Program signage

Double Up Oklahoma promotional signs rarely had quality issues and were consistently displayed in produce departments in both program years. However, signage in other areas of stores was inconsistently displayed and declined from PY01 to PY02 (Figure 5). Accordingly, store personnel expressed mixed feedback regarding the effectiveness of signage, including perceptions of diminishing effectiveness over time (Box 2). Produce managers conveyed a preference for the large size and visibility of Iron Man posters and described difficulty in maintaining the placement of the notecard-sized nudge cards. When one or more nudge cards were observed (36.9% of site visits), over seven nudge cards were present on average.

Cashier-customer interactions

Of the 923 observed cashier transactions, 491 (53.2%) involved SNAP. During these transactions, just over one-third of cashiers informed customers of DUO earnings in PY01 (36.8%) and PY02 (35.4%). Observations of other interpersonal, promotional activities were lower and declined from PY01 to PY02 (Table 3); interviews with staff indicated these ongoing promotions were perceived as unnecessary since repeated customers in small, rural communities became familiar with DUO over time. However, assistant managers identified direct initial communications with customers as the premier approach to promoting DUO, as they contend that a positive experience within the store may result in recommendations throughout the community (Box 2).

Facilitators and barriers to DUO implementation and utilization

Key informant interviews were critical in understanding how both customers and store personnel interacted with the program.

Transaction-related barriers to implementation

Box 3 provides a detailed overview of transaction-related barriers to implementation commonly reported by store personnel, including issues related to the vouchers' thermal paper (Figure 6), processing of multiple payment sources within the same transaction, and the daily earning limit.

Reported customer feedback: Program barriers, facilitators, and perceived benefits

Reports of barriers to customer utilization were uncommon and declined over time. Store personnel reported confusion among some customers about F/V eligibility, specifically

Box 2. SELECTED QUOTATIONS FROM KEY INFORMANT INTERVIEWS WITH GROCERY STORE PERSONNEL WORKING IN DIFFERENT EMPLOYEE ROLES DURING FORMATIVE EVALUATION SITE VISITS FOR THE DOUBLE UP OKLAHOMA (DUO) PROGRAM

Figure 5.

Fidelity to display of Double Up Oklahoma (DUO) nutrition incentive program poster signage across 10 grocery stores, by program year (total N=228 site visits).

Table 3.

CASHIER FIDELITY TO DOUBLE UP OKLAHOMA (DUO) PROMOTIONAL ACTIVITIES AT THE POINT OF SALE DURING SUPPLEMENTAL NUTRITION ASSISTANCE PROGRAM (SNAP) TRANSACTIONS, BY PROGRAM YEAR (TOTAL N=491 TRANSACTIONS).

canned or frozen F/Vs, as well as salad kits that include DUO-ineligible ingredients, such as dressings or other non-produce food items. As one assistant manager remarked, "[Incidents of confusion have] tapered as people become familiar with the program."

Store personnel conveyed overwhelmingly positive DUO feedback received from customers, often within three categories: reduction of financial barriers, empowerment in attaining personal health goals, and improved ability to promote healthier dietary habits within the household (Figure 7).

Lessons Learned

Nutrition incentive programs face implementation challenges specific to grocery store settings, and our on-site evaluation was critical in identifying emergent issues in a timely manner as part of the developmental evaluation aspect of our formative evaluation. The findings from the process evaluation provide three key lessons: 1) the importance of knowledgeable employees as primary program promoters with a focus on routine trainings, 2) anticipation of increased personnel needs to accommodate new produce demand attributable to NIP-generated sales, and 3) the necessity of regular communication between the NIP awardee and collaborating food retailers to ensure optimal customer experiences and program execution.

A primary limitation of our formative evaluation was reliance on staff member perceptions and recollections of customer feedback; future evaluations should incorporate direct interaction with customers to ensure their perspectives are adequately captured. However, the strengths of this multi-faceted formative evaluation include soliciting

Box 3. BARRIERS TO IMPLEMENTATION OF THE DOUBLE UP OKLAHOMA (DUO) NUTRITION INCENTIVE PROGRAM REPORTED BY GROCERY STORE PERSONNEL AND RESOLUTIONS ACHIEVED THROUGH FORMATIVE EVALUATION

Figure 6.

An identified barrier to Double Up Oklahoma (DUO) nutrition incentive program participation included vouchers with unscannable barcode and partially unreadable serial number (example shown).

Figure 7.

Summary of themes with illustrative quotations resulting from key informant interviews with grocery store personnel that describe the potential customer-focused benefits of the Double Up Oklahoma (DUO) nutrition incentive program.

multiple perspectives from store personnel and providing rapid feedback to evaluation users that improved implementation and customers' experiences. Additionally, future research may benefit from exploring the impact of NIP on the operational needs of host grocery stores, including produce department staffing needs, changes in inventory and ordering processes, and changes to the physical produce food environment. Possible consumer-level outcome evaluations could further explore the impact of NIP on adequacy of food resources as well as health behaviors within the household to allow for a more holistic understanding of the roles of these programs in building the resilience of communities.