Children playing with pictures of fruit

Children playing with pictures of fruit


View the PDF Version

Key findings

Outreach by text increased awards of Minnesota’s Child Care Stabilization Base Grants (CCSBG) by 3.8 percentage points, while combining calls with text outreach increased awards by 5.2 percentage points. Among child care providers with no prior awards, outreach that included calls increased awards by 12.4 percentage points — an effect five times larger than outreach by text alone. In contrast, among child care providers previously awarded grants, outreach by text alone increased awards by 5.3 percentage points.

Agency priority

The COVID-19 pandemic highlighted the fragility of the child care market and placed additional financial burdens on child care providers.1 Child care is essential for American communities to thrive, but the system’s current structure means many families cannot access or afford high-quality care, and the workforce is underpaid for skilled and valuable work.2 Stabilizing the child care market and providing higher wages to child care workers has important equity implications, as nearly all child care providers are small businesses, are overwhelmingly owned by women, and are disproportionately owned by people of color.3

In response to these challenges, the American Rescue Plan (ARP) allocated approximately $24 billion in grant funding to help stabilize the child care labor market.4 The Department of Health and Human Services (HHS)’s Administration for Children and Families (ACF) worked with states, territories, and Tribes to distribute stabilization sub-grants to child care providers. The Minnesota Department of Human Services (MN DHS) implemented the program in Minnesota and awarded CCSBG grants of $400 per full-time equivalent employee each month.5 Recent research at the national level found suggestive evidence that CCSBG’s were effective at increasing child care employment and wages.6 However, about a third of child care providers in Minnesota did not receive a CCSBG award or declined their funding from September 2021 to January 2022, underlining the need to enhance equitable access to CCSBG among child care providers who had never or infrequently been awarded the grant.

Program change description

Prior research has shown that proactive outreach through calls, texts, and emails improves the take-up of similar social benefits programs by simplifying information and providing reminders and planning prompts.7,8 Yet, providers who had never been awarded grants faced additional barriers to applying for CCSBG. A MN DHS survey indicated that this key priority group for enhancing equitable access held misconceptions about the application process, the benefits of funding, and their eligibility.

We worked with MN DHS to design a communications bundle that consisted of semi-scripted calls, one-way text messages, and modified emails. The multimodal communications bundle was designed to reach potential beneficiaries who might not be reached via one mode of contact, but could potentially be reached through a different mode, and included specific information designed to address providers’ application misconceptions. We directed a higher proportion of calls to providers with no prior awards since we believed that a conversation with a staff member would be necessary to address their concerns and induce this priority group to apply.9

Evaluation design

MN DHS implemented the interventions monthly for five months from February to June 2023. At the outset, we randomized 3,265 likely eligible child care providers who had missed at least one award in the prior six months to business-as-usual, text, or call groups.10 MN DHS sent providers in the business-as-usual group a standard set of emails with information encouraging them to apply when the monthly application period opened, and two email reminders before it closed. In addition to the monthly email announcing that the application window had opened, MN DHS sent the text group a modified application opening email, a text reminder, and two modified reminder emails before the application period closed. MN DHS sent the call group the same communications bundle as the text group each month, and a MN DHS staff member called them once over the course of the evaluation.

One of the three versions of communications were sent to all eligible child care providers, and no changes were made to the application review process or requirements. Half of the providers were randomized to the business-as-usual group (n=1,636), and the other half to the text (n=735) or call (n=894) group. The evaluation had a stepped wedge design for the text and call groups.11 Randomization occurred within blocks, accounting for provider type (family-based care vs. child care centers), their prior award history, and the content of an opt-out text that providers received as part of another evaluation (see Learning about transparent defaults in opt out text messages). Providers who were never awarded the grant or were awarded the grant inconsistently over the prior six months were more likely to be randomized to the call than the text group than providers who had missed only one award.12

Analysis of existing data

MN DHS collected monthly CCSBG data detailing which providers received awards and the amount of funding they received. The primary outcome for this evaluation was whether a provider received funding in a given month. We evaluated the effect of texts and calls compared to business-as-usual, as well as the difference in the effects of texts and calls, on whether providers were awarded funding each month.13 In exploratory analysis, we disaggregated the effect by award history.

Results

Our primary analysis found that both the texts and calls increased CCSBG awards compared to the business-as-usual group, in which 37.3% of providers were awarded grants each month. Texts increased the proportion of providers receiving awards by 3.8 percentage points per month (p < 0.01, 95% CI [1.6, 6.1]). Calls increased the proportion of providers receiving awards by 5.2 percentage points per month (p = 0.02, 95% CI [0.7, 9.6]).14 The difference between these effects was not statistically significant (p=0.55).

Figure 1. Texts and calls increased the monthly proportion of providers that were awarded CCSBG Bar chart showing that texts and calls increased the monthly proportion of providers that were awarded CCSBG Note: + p < 0.1, * p < 0.05, ** p < 0.01 (two-tailed) when comparing text or call to business as usual.

Calls were particularly effective among providers who had no prior awards. Among this subgroup, providers who were called were three times as likely to be awarded grants compared to providers in the business-as-usual group (18.5% in the call group compared to 6.1% in the business-as-usual group). Moreover, for this subgroup, the 12.4 percentage point effect of calls (p < 0.01, 95% CI [8.3,16.5]) was over five times as large as the 2.2 percentage point effect of texts (p = 0.11, 95% CI [-0.5,5.2]), a statistically significant difference in effects (p < 0.01).

In contrast, texts were sufficient for increasing awards among providers who had prior awards. Among this subgroup, providers in the text group were 5.3 percentage points more likely to be awarded grants (p < 0.01, 95% CI [1.9, 8.7]) than providers in the business-as-usual group (73.5% in the text group compared to 68.1% in the business-as-usual group). The effect of calls for these providers was imprecisely measured and not statistically significant (-1.7 percentage points, p < 0.71, 95% CI [-11.2, 7.7]) nor statistically distinguishable from the effects of texts (p = 0.13).

Figure 2. Calls were more effective for providers without previous awards, whereas texts were more effective for providers with previous awards Bar chart showing that Calls were more effective for providers without previous awards, whereas texts were more effective for providers with previous awards Note: + p < 0.1, * p < 0.05, ** p < 0.01 (two-tailed) when comparing text or call to business as usual.

Implications

We found that both texts and calls had a large positive effect on the probability of CCSBG award funding for child care providers, a group disproportionately composed of underserved individuals. Calls were particularly effective among providers without prior awards, whereas texts alone were sufficient to increase awards among providers previously awarded grants. Given the effectiveness of CCSBG in stabilizing the child care market as a whole, it is possible that increasing grant awards among providers would reduce closure rates.15

Costs for calls were comparable to other interventions promoting applications or enrollment across a range of contexts, while texts were more cost effective on average.16 The variable cost per additional provider who was awarded funding was $23.07 for those in the call group and $0.22 for those in the text group.17 For those without prior awards, costs for calls were $10.56 per additional provider awarded funding. For those with prior awards, costs for texts were $0.16 per additional provider awarded funding. The cost differential is primarily due to the staff time necessary to make calls.18

Other agencies looking to increase the rates of benefit awards could consider incorporating both texts and calls, but should carefully consider their targeting. Although calls were less cost-effective than texts for the full sample of providers, they closed important gaps in receipt of CCSBG and were necessary to generate significant increases in awards among those least likely to otherwise receive funding. Texts were effective and less costly, but exacerbated pre-existing disparities in award rates.

Taken together, it is likely that texts, which are low cost, work best for those already predisposed to apply, whereas calls, a higher-cost, higher-touch intervention, are likely necessary to generate significant changes in behavior. More generally, the findings show that specific types of interventions work for specific populations, and underline the importance of understanding which interventions are likely to address the specific behavioral barriers faced by priority subgroups in the target population.

Notes:

  1. Administration for Children and Families, “Information Memorandum ARP Act Child Care Stabilization Funds.”
  2. Ibid.
  3. White House Fact Sheet “FACT SHEET: Biden-⁠Harris Administration Announces American Rescue Plan Funding to Rescue the Child Care Industry so the Economy Can Recover.”
  4. Administration for Children and Families, “Information Memorandum ARP Act Child Care Stabilization Funds.”
  5. Between February and June 2023, the CCSBG amount varied from $205 to $400 per full-time equivalent employee.
  6. Council of Economic Advisors Working Paper, “Did stabilization funds help mothers get back to work after the COVID-19 recession?” November 7, 2023.
  7. Castleman, Benjamin L., and Lindsay C. Page. “Freshman year financial aid nudges: An experiment to increase FAFSA renewal and college persistence.” Journal of Human Resources 51, no. 2 (2016): 389-415.
  8. Page, Lindsay C., Bruce I. Sacerdote, Sara Goldrick-Rab, and Benjamin L. Castleman. “Financial aid nudges: A national experiment with informational interventions.” Educational Evaluation and Policy Analysis (2022): 01623737221111403.
  9. About a quarter of calls were answered. Once a phone call was answered, the most frequently discussed topics with a MN DHS staff member were eligibility, allowable uses, and program benefits.
  10. A total of 3,374 child care sites made up the 3,265 providers. Sites were clustered together if they shared a name, location, or contact information.
  11. With a stepped wedge design providers were considered as being called in all months after they were called, so over time more providers were in the call group. The call order was randomized.
  12. The percentage of providers randomized to the call group was 43% for providers never awarded a grant, 14% for providers awarded two to four grants, and 17% for providers awarded five grants in the prior six months.
  13. All analyses included weights to account for the differential probability that providers were assigned to the call group, and included clustering by provider address and phone number. We controlled for the month in which the groups were assigned to the intervention, whether they were located in a rural area at baseline, and their assignment block.
  14. The preregistered familywise error rate threshold is estimated to be 0.013, indicating this result falls just short of statistical significance.
  15. Council of Economic Advisors Working Paper, “Did stabilization funds help mothers get back to work after the COVID-19 recession?” November 7, 2023.
  16. See for example Weixler, Lindsay, et al. “Helping parents navigate the early childhood education enrollment process: Experimental evidence from New Orleans.” Educational Evaluation and Policy Analysis 42.3 (2020): 307-330; Abdul Latif Jameel Poverty Action Lab (J-PAL). 2023; “The effect of nudges on health insurance take-up in the United States” J-PAL Policy Insights. Last modified April 2023. https://www.povertyactionlab.org/policy-insight/effect-nudges-health-insurance-take-united-states; and Patterson, Richard W., and William L. Skimmyhorn. How do behavioral approaches to increase savings compare? Evidence from multiple interventions in the US Army. No. w30697. National Bureau of Economic Research, 2022.
  17. Cost effectiveness analyses include the direct additional costs that DHS incurred as a result of the outreach. Costs of texts included the sending cost of texts. Costs for calls include costs for calls and texts, since texts were sent to the call group as well. The call costs only include hours logged making calls and documenting them, and not billed hours due to concerns with measurement error. We assumed that there is no cost to make an additional call from on a landline.
  18. We calculate only cost-effectiveness and not uncertainty from noise in the effect sizes or costs. Estimates are calculated by taking the cost of implementation for the intervention group and dividing it by the additional number of providers that were induced to receive an award as a result of the intervention.