Evaluating the impact of communications on online account creation and case linkage

Person using a phone

Person using a phone

What is the agency priority?

The U.S. Department of Homeland Security’s U.S. Citizenship and Immigration Services (USCIS) aims to increase customer satisfaction by encouraging the effective and efficient use of USCIS’s online services and contact centers as part of their mission to uphold America’s promise as a nation of welcome and possibility with fairness, integrity, and respect for all they serve. Increasing the number of customers who use USCIS’ online portal, myUSCIS, is a key agency priority as part of their efforts to reduce the paperwork burden on customers and agency staff.

What did we evaluate?

Customers may underutilize online tools for a variety of reasons, including a lack of information, a (mis)belief that such tools are not secure, or a perception that using online tools is cumbersome.1 As part of its existing outreach efforts, USCIS sends a one-time email to customers who have recently filed a form by mail, encouraging them to create an online account and link their paper- filed case to their account. We evaluated two alternate email messages against the USCIS status quo message to understand which message framings are most effective at encouraging customers to engage with online services. The first message (operational transparency message) focused on the benefits of creating an online account to track one’s case in real time, while the second (social norms message) emphasized that millions of other customers had already signed up for online accounts.

How did the evaluation work?

We randomly assigned 261,398 customers to one of three groups: 1. the status quo group, assigned to receive the USCIS’s existing outreach email (n=86,469); 2. the operational transparency group, assigned to receive the simplified email that emphasized the benefits of using online accounts to track one’s case status (n=87,702); and 3. the social norms group, assigned to receive the simplified email that emphasized that many other customers had already signed up for online accounts (n=87,227).

Customers were clustered at the email address level, and each email was randomly assigned to a message group. New customers were enrolled into the intervention once per month over the course of three months (May, July, and August 2023).

What did we learn?

We intended to analyze the impact of the randomized outreach using existing USCIS data on the creation of online accounts and the linkage of paper-filed cases to online accounts. We were not able to complete the analysis as planned. In cases where data are not available or the evaluation did not provide comparable comparison groups, we do not report results. In this case, the evaluation team learned that customers who received outreach already had online accounts, but needed to activate them, leading to a mismatch between the requested action and the necessary behavior. This evaluation demonstrated the feasibility of embedding a rigorous evaluation into ongoing outreach efforts at USCIS. It also enhanced cross-team collaboration within USCIS, identified areas in which ongoing informational outreach could be modified to be better aligned with organizational processes, and suggested new avenues for evaluating engagement with online services in the future.

Notes:

  1. Herd, P., & Moynihan, D. P. (2019). Administrative Burden: Policymaking by Other Means. Russell Sage Foundation. https://doi.org/10.7758/9781610448789.

Verify the upload date of our analysis plan on GitHub.

Year

2024

Status

Complete

Project Type

Impact evaluation of program change

Agency

Homeland Security

Domain

Communications

Resources

View Analysis Plan (PDF) View Abstract