Sign that says we welcome SNAP benefits

Sign that says we welcome SNAP benefits


View the PDF Version

Key findings

A text campaign that included a link to a redesigned document uploader resulted in fewer applicants being denied Supplemental Nutrition Assistance Program (SNAP) benefits due to incomplete eligibility information by 1.1 percentage points. The intervention did not have a meaningful impact on SNAP enrollment.

Agency priority

The Supplemental Nutrition Assistance Program (SNAP), administered by the U.S. Department of Agriculture (USDA), provides food assistance to 1 in 8 Americans, or over 42 million people annually.1 However, participation rates vary considerably across states and an estimated 7 to 8 million additional Americans are eligible for SNAP, but are not enrolled in the program.2 The American Rescue Plan Act of 2021 (ARP) allocated over $1.1 billion to USDA to help states administer SNAP in response to increased demand for food benefits during the pandemic.3 Most states used at least part of these funds to invest in technology modernization as a way to improve customer experience with SNAP and streamline the program’s application process.4 Through this modernization effort, USDA seeks to increase food security and ensure that SNAP is accessible to the communities it is intended to serve.5

Program change description

Past research finds that individuals often fail to take up public benefits programs due to administrative burdens, such as the learning costs of obtaining relevant program information and compliance costs of completing time- and effort-intensive application activities.6,7,8 Applying for SNAP benefits in the collaborating state requires applicants to submit applications, provide the state with multiple verification documents, and complete an interview with a caseworker. Failure to complete the final two steps results in a denial of benefits due to incomplete information rather than a denial due to ineligibility.

While the convenience of online application and recertification portals has increased participation in public benefits programs like SNAP, these technologies can introduce burden when they require applicants to navigate complex tasks and confusing web interfaces unassisted.9,10 For instance, uploading verification documents into an online system often requires applicants to remember their login information and can create uncertainty when applicants must keep track of which documents they have and have not uploaded. With these barriers in mind, Code for America (CfA) collaborated with a large southern U.S. state to create an easy-to-use, mobile-friendly, web-based document uploader to make it easier for applicants to submit verification documents.11 CfA’s redesigned document uploader improves upon the existing uploader by leveraging human-centered design principles, eliminating cognitively-demanding password requirements, and increasing the uploader’s usability on mobile devices.12,13,14

Evaluation design

A text campaign encouraging the use of the redesigned document uploader was evaluated using an applicant-level randomized evaluation.15 All applicants were sent up to seven text messages reminding them to upload documents between the time they submitted their application and the application deadline.16 However, in practice, most applicants were sent only the first two messages.17 Between September and December 2023, online SNAP applicants (N=33,574) who applied as part of the regular online process were randomized to a basic text message group (n=16,653) or uploader text message group (n=16,921). For applicants assigned to the uploader text group, these text messages included individualized links which granted them access to the redesigned document uploader and encouraged its use. The redesigned document uploader included two key features: it did not require a log-in with email and password, and it allowed applicants to upload documents by taking a photo or dragging and dropping files from their desktop. No changes were made to the SNAP application or approval process beyond the introduction of this new, alternative method for upload verification documents and the introduction of the text campaign.18

Analysis of existing data

Operational data and application responses from the collaborating state and CfA were used to compare outcomes between applicants assigned to the basic text and uploader text groups and to monitor implementation of the intervention.19 These data captured the primary outcomes for the evaluation: whether the application was denied due to incomplete information and whether the applicant received any SNAP benefits, a measure of SNAP enrollment.20 Additionally, the data captured demographic and household characteristics, when applicants submit their initial application, and implementation measures such as text message delivery status, and whether an applicant submitted documents via the redesigned document uploader.

Results

Our primary analysis found that while the uploader text messages decreased denials due to incomplete information, encouragement to use the redesigned document uploader had no effect on SNAP enrollment rates.21 Data constraints limited our ability to measure the effects of the intervention on our primary outcome of whether verification documents were uploaded via any method.22 However, we can describe the extent to which the text message campaign encouraged use of the redesigned document uploader. Even though the majority of applicants were sent only two text messages (rather than the planned seven), 30.6% of applicants in the uploader text group used the redesigned uploader to upload at least one document. In the basic text message group 21.1% of applicants were denied SNAP benefits because caseworkers did not have enough information to evaluate whether they were eligible. Applicants in the uploader text group were 1.1 percentage points less likely to be denied due to incomplete information (p = 0.012, 95% CI [-0.21, -1.90].

Figure 1. Encouragement to use the redesigned document uploader reduced SNAP denials due to incomplete information Bar chart showing encouragement to use the redesigned document uploader reduced SNAP denials due to incomplete information

Although encouragement to use the redesigned document uploader decreased denials due to missing information, it ultimately did not increase enrollment in SNAP. Among applicants in the basic text group, 43.3% enrolled in SNAP, compared to 43.8% of applicants in the uploader text group. This difference of 0.5 percentage points is not statistically significant (p = 0.28, 95% CI [-0.46, 0.15]).

Figure 2. Encouragement to use the redesigned document uploader had no detectable effect on SNAP enrollment Bar chart showing encouragement to use the redesigned document uploader had no detectable effect on SNAP enrollment

Implications

A text message encouragement campaign to use a redesigned document uploader had no detectable effect on enrollment in SNAP, but decreased denials due to incomplete eligibility information. This decrease means that caseworkers were able to determine eligibility for more applicants who were sent messages about the redesigned document uploader than applicants who were sent simple reminder messages.

One explanation for these results is that some applicants who were encouraged to use the redesigned document uploader were ultimately ineligible for SNAP. These applicants may never have followed through with uploading documents under a more onerous system, but since the texts made it easier for them, they submitted their documents, but then learned they were ineligible or withdrew their application.23

Second, although the intervention delivery was perhaps weaker than it could have been — amounting to just two text messages encouraging the use of the redesigned document uploader — we observed meaningful uptake of the redesigned document uploader, with limited spillover into the control group.24 Given the uploader usage rates, it is unlikely that the null effect on SNAP enrollment is simply driven by a lack of people clicking on the uploader link in the text message.

Data constraints limited our ability to measure the impact of the intervention on uploading verification documents, the behavior that the intervention was more directly designed to encourage. Isolating the effect of the intervention on document upload rates would help disentangle the degree to which uploading any documents is a meaningful behavioral barrier to SNAP enrollments. This would also help identify whether other barriers, including uploading any documents at all, or attending a SNAP interview, may need to be addressed through other interventions.

Moreover, because this evaluation relies on administrative data linked to application outcomes, we can directly measure whether applicants successfully navigated steps in the application process and their enrollment determination. Other measures, such as customer experience surveys, would be better suited to detect whether the redesigned document uploader facilitated a better customer experience during the SNAP application process.25

Finally, the bundled intervention design (a text-encouragement campaign that both made the redesigned document uploader more readily accessible by providing applicants a direct link and featured a redesigned uploader) makes it difficult to disentangle the effects of access to the redesigned uploader from the effect of the uploader itself. Future evaluators could consider the feasibility of alternative designs that help disentangle these effects. For example, including a link to the existing method for uploading documents in the basic text group or adding a third group that does not receive outreach would generate comparisons that help understand the mechanisms behind the bundled intervention evaluated here. These designs were deemed infeasible to implement at the time of fielding this evaluation.

Improving the customer experience and the application process has been a focus for many ARP-funded SNAP modernization projects.26 Other interventions may be more effective at moving the needle on priority outcomes than the one evaluated in this study. For instance, policymakers could consider alternative intervention approaches that couple human-centered design changes to existing processes — a focus of this intervention — with deeper modifications to how the program operates, such as interventions that modify when applicants are able to schedule interviews for SNAP verification.27

Notes:

  1. “Program Data Featured Reports, Fiscal Year 2023” U.S. Department of Agriculture Food and Nutrition Service, accessed July 18, 1014. https://www.fns.usda.gov/pd/overview.
  2. “Strategies for Reducing Administrative Burden in Public Benefit and Service Programs” Office of Management and Budget, accessed July 28, 2024, https://www.whitehouse.gov/wp-content/uploads/2022/12/BurdenReductionStrategies.pdf.
  3. “American Rescue Plan Fact Sheet” U.S. Department of Agriculture, accessed July 28, 2024, https://www.usda.gov/sites/default/files/documents/arp-national-factsheet.pdf.
  4. “Exploring States’ SNAP Modernization Projects” Urban Institute, accessed July 28, 2024, https://www.urban.org/projects/exploring-states-snap-modernization-projects.
  5. Strategic Plan Fiscal Years 2022 – 2026, p. 32.” U.S. Department of Agriculture, accessed July 18, 2024, https://www.usda.gov/sites/default/files/documents/usda-fy-2022-2026-strategic-plan.pdf
  6. In the administrative burden framework, costs of benefits take up also are decomposed into a third category: the psychological costs of participating in a program that may carry stigma or result in a loss of autonomy.
  7. Moynihan, Donald, Pamela Herd, and Hope Harvey. “Administrative burden: Learning, psychological, and compliance costs in citizen-state interactions.” Journal of Public Administration Research and Theory 25, no. 1 (2015): 43-69.
  8. Bertrand, M., Mullainathan, S., & Shafir, E. (2006). Behavioral economics and marketing in aid of decision making among the poor. Journal of Public Policy & Marketing, 25(1), 8-23.
  9. Gray, Colin. “Why leave benefits on the table? Evidence from SNAP.” (2018): 18-288.
  10. Madsen, Christian Østergaard, Ida Lindgren, and Ulf Melin. “The accidental caseworker–How digital self-service influences citizens’ administrative burden.” Government Information Quarterly 39, no. 1 (2022): 101653.
  11. As part of the redesigned document uploader, CfA implemented a Robotic Processing Automation to submit the documents to the state agency that administers the SNAP program. When applicants uploaded documents using CfA’s redesigned uploader, an RPA process would then submit documents to the state agency. Caseworkers then could access the pool of submitted documents and manually match SNAP applicants to the untagged documents.
  12. Nielsen, Jakob. “Enhancing the explanatory power of usability heuristics.” In Proceedings of the SIGCHI conference on Human Factors in Computing Systems, pp. 152-158. 1994.
  13. Inglesant, Philip G., and M. Angela Sasse. “The true cost of unusable password policies: password use in the wild.” In Proceedings of the sigchi conference on human factors in computing systems, pp. 383-392. 2010.
  14. Toepoel, Vera, and Peter Lugtig. “What happens if you offer a mobile option to your web panel? Evidence from a probability-based panel of internet users.” Social Science Computer Review 32, no. 4 (2014): 544-560.
  15. Applications associated with applicant households were randomized to the basic text or uploader text groups based on the timestamp associated with when they submitted their initial online application.
  16. The text messages campaign included messages scheduled to be sent after the initial application was submitted, before the interview, seven days before the deadline, one day before the deadline, and on the deadline. For the basic text group, the text message before the interview included a list of types of verification documents that applicants may need to submit.
  17. Due to a technical error, 82.9% of the sample were sent the first two messages and 18.1% of the sample were sent the full series of messages. A similar share of applicants in the basic text message group and the uploader text message group were affected by this error.
  18. All applicants had access to the other modes for submitting verification documents, which included submitting documents through a state-operated online portal, or submitting documents via mail, email, fax, or in person.
  19. Unless noted otherwise, all of the analysis reported in this abstract was prespecified in an analysis plan, which can be found at https://oes.gsa.gov/2310-decreasing-snap-denial-rates/.
  20. Due to data limitation, we were unable to measure our third primary outcome whether verification documents were uploaded using any method and linked to the application.
  21. All analyses included Lin-adjusted controls for the week of application submission, applicant age and household size, zip code median income, and indicator variables for whether the applicant was a person of color, applied for benefits via a non-English language option, or lived in a rural zip code.
  22. We pre-specified a multiple hypothesis correction controlling for the family-wise error rate (FWER) through simulation, but chose instead to use a Bonferroni correction for our two primary outcomes. The significance of the findings do not change after adjusting for multiple hypotheses using the Bonferroni correction.
  23. Exploratory analysis suggests that applicants in the uploader text group were more likely to be found ineligible, though the difference in ineligibility rates is not statistically significant. Applicants in the uploader text group were 0.53 percentage points more likely to be found ineligible (p = 0.25, 95% CI [-0.38, 1.45]).
  24. In the uploader text group, nearly half of applicants (48.3%) interacted with the redesigned document uploader in some way and nearly a third (30.6%) used the redesigned document uploader to share one or more documents. In the basic text group, less than one percent of applicants interacted with the redesigned uploader (0.7%) or used it to upload a document (0.6%).
  25. The Better Government Lab at Georgetown University analyzed a complementary customer experience survey in collaboration with Code for America as part of the evaluation of the redesigned uploader.
  26. “Exploring States’ SNAP Modernization Projects.” 2023 Urban Institute, accessed July 28, 2024, https://www.urban.org/projects/exploring-states-snap-modernization-projects.
  27. Giannella, Eric., T. Homonoff, G. Rino, and J. Somerville. “Administrative Burden and Procedural Denails: Experimental Evidence from SNAP.” American Economic Journal: Economic Policy. Forthcoming.