Measuring delivery and opt out-rates for a text message campaign with a state partner

Person on an iphone

Person on an iphone

What is the agency priority?

As individuals increasingly rely on digital technologies to navigate their day-to-day lives, government agencies have made it a priority to incorporate them into their service delivery models. The Department of Health and Human Services’ Administration for Children and Families and Minnesota’s Department of Human Services (MN DHS) wanted to use text messages to send eligible child care providers information and reminders about their Child Care Stabilization Base Grant (CCSBG) program, implemented as part of the American Rescue Plan. MN DHS was required by a recent Federal Communications Commission ruling to give text message recipients the option to opt out of text message outreach. MN DHS sought ways to increase the share of providers who continued to receive messages so they could keep providers informed of priority program updates.

What did we evaluate?

We worked with MN DHS to leverage the power of defaults in their text message outreach by enrolling providers into their texting program by default. Default options are options that are initiated automatically and do not require additional steps to pursue. Because it takes more effort to opt out of a default than to agree with one, defaults are powerful tools in promoting desired behaviors. How the default option is presented, however, may influence the likelihood that an individual opts out of that choice. Transparent defaults draw attention to the fact that a default option has been selected and explain how the default might impact someone who agrees with it — making default options seem more trustworthy and compelling.

How did the evaluation work?

We randomized the version of the opt-out message sent to 3,374 childcare providers (associated with 3,268 phone numbers) who were identified in state licensure data as likely eligible for CCSBG, and evaluated the effect of a transparent default message on providers’ continued enrollment in MN DHS’ texting program. Three days after the February CCSBG application period opened (February 17, 2023), MN DHS sent half the providers in the evaluation the transparent message and the other half the standard message.

What did we learn?

We saw no difference in opt-out rates between the standard and transparent message groups. However, only a third of the text messages sent were successfully delivered. A large volume of delivery failures and low opt-out rates in both groups limited our ability to detect small differences in opt-out rates between these groups. Among providers who had text messages successfully delivered, less than 3% opted out of the default option to receive future text messages.

Although text messages can be an effective tool to communicate with large numbers of individuals, throttling and other factors can affect whether outgoing text messages reach their intended audience. The low opt-out rate among providers who had messages delivered corroborates a well-established finding that individuals conform to defaults. Asking individuals to opt out of text messages rather than opt into them can increase subscription rates, enabling the sharing of information on important public benefits.

Figure 1. Transparent defaults did not affect opt out rates or successful delivery of messages, but throttling was greater in the transparent message group Bar chart showing that transparent defaults did not affect opt out rates or successful delivery of messages, but throttling was greater in the transparent message group

Verify the upload date of our Analysis Plan on Github.

Year

2024

Status

Complete

Project Type

Impact evaluation of program change

Agency

Health and Human Services

Domain

Communications, Pandemic Relief and Recovery

Resources

View Analysis Plan (PDF) View Abstract View Intervention Pack (PDF)