Increasing applications to the Homeowner Assistance Fund using mailed outreach

View the pdf version

Key findings

We find strong evidence that postcards did not increase the number of applications to Idaho’s Homeowner Assistance Fund. The findings enabled IHFA to redirect approximately $25,000 to fund English and Spanish language billboards targeting eligible homeowners in hard-to-reach rural areas with above average shares of Hispanic/Latino populations.

Target a priority outcome

The Homeowner Assistance Fund (HAF) program, overseen by the U.S. Department of the Treasury (Treasury), provides $9.96 billion in funding to states, territories, and Tribal entities (grantees) to prevent the displacement of homeowners as a result of the COVID-19 pandemic. HAF provides funds to eligible homeowners to assist with mortgage payments and other qualified expenses related to housing.1 Treasury encouraged grantees to use targeted outreach to maximize program accessibility, especially among low-income and socially disadvantaged homeowners.2

The evaluation described here contributes to the American Rescue Plan Equity Learning Agenda question on awareness of and access to ARP funds, including among underserved individuals and communities, as well as Treasury’s Office of Recovery Programs Learning Agenda.

Translate evidence-based insights

The Idaho Housing and Finance Association (IHFA) launched its HAF program in June 2022. The program offered up to $50,000 in mortgage assistance to eligible homeowners facing financial hardship associated with the COVID-19 pandemic. Applications were submitted using a form on IHFA’s website. IHFA planned email, social media, and print mail campaigns to increase program applications. IHFA previously used all of these outreach methods when running its Emergency Rental Assistance program and believed print mail could reach households at risk of foreclosure who could not be targeted via email or social media.

However, outreach via print mail is relatively costly and labor-intensive. Evidence from our past randomized evaluations suggests that targeted outreach via mail can have mixed results in increasing the uptake of social programs.3 Before sending the print mail to a planned 26,000 households, we collaborated with IHFA to understand whether initial direct mail efforts were effective at increasing uptake of their HAF program among a small sample of recipients.

Embed evaluation

IHFA designed a postcard with information on program eligibility and how to submit an application printed on the front and back. We designed and implemented a randomized evaluation to measure the impact of mailing that postcard on applications to IHFA’s HAF program.4 The postcard targeted a sample of 1,651 homeowners in pre-foreclosure or foreclosure. The mailing list was compiled by a third-party vendor. The evaluation was powered to detect increases of 0.2 percentage points or more. In December 2022, 826 addresses were randomly assigned to be sent the mailer early and 825 to be sent four weeks later.5

Analyze using existing data

The primary outcome was the application initiation rate, defined as having filled in basic information about the applicant’s name and address on the application form.6 Secondary outcomes were application completion rates, funding status, and comparison of applicant demographic characteristics (race, gender, ethnicity, income). Data was collected at two time points through IHFA’s application vendor, once before the early mailer was sent and again four weeks after the early mailer was sent.

Results

Application initiations and completions were very low for both groups: there were three initiated applications and two completed applications in the early mailer group, compared to one initiated application and one completed application in the holdout group.7 The statistical estimates suggest that the postcards had no effect.

Four weeks after the early mailers went out, the group that had not yet been sent the postcard had an application initiation rate of one-tenth of a percent (0.12%). The group who had been sent the postcard four weeks prior had an application initiation rate of roughly one-third of a percent (0.37%). This difference of a fraction of a percentage point (0.24 or the equivalent of 3 households) is substantively small and statistically insignificant (p = 0.327, 95% CI [-0.24, 0.73]).8 This is not an imprecise null: we can reject the hypothesis that the effect of the mailers on application initiations is any greater than 0.72 percentage points with 95% confidence.9 The same result holds for application completion rates: the early mailer group had a statistically insignificant 0.12 percentage point greater completion rate (p = 0.563, 95% CI [-0.25, 0.50]).

Due to data availability, we were unable to run our planned analyses of effects on funding status.10 We were unable to complete the pre-registered demographic test, as the low number of initiated applications meant that this was not estimable. However, given that the application initiation rate does not appear to be affected by the postcard, we have no reason to believe these outcomes would either.

Figure 1. Percentage of households that initiated HAF applications Percentage of households that initiated HAF applications

Build evidence

This evaluation provided rapid cycle evidence that informed decision-making by IHFA. From design and implementation of the randomization in November 2022, to fielding, analysis, reanalysis, and delivering the results to IHFA in March 2023, the entire study took us less than five months to implement. IHFA had planned to send a second, larger postcard campaign to approximately 26,000 households in minority-majority census tracts. Because those households were not identified based on foreclosure risk, it seemed even less likely the postcard would be effective if sent to them. Based on our rapid delivery of the results, IHFA decided not to move forward with scaling up the postcard. Those funds (approximately $25,000) were redirected to fund English and Spanish language billboards targeting eligible homeowners in hard-to-reach rural areas with above average shares of Hispanic/Latino populations.

This evaluation has implications for other state housing agencies’ outreach strategies: postcards may not be the most effective way to engage homeowners who are eligible for HAF, even those who are most in need. In this evaluation, our analyses were well-powered to detect the effects of mailed outreach to high-risk homeowners — where HAF funding could prevent them from losing their homes. We find that postcards did not impact their likelihood to initiate or complete applications. Future work could examine the effectiveness of more intensive interventions — for example, phone outreach, in-person outreach, or dedicated customer support to help homeowners complete applications.

Notes:

  1. See HAF guidance for a full list of qualified program expenses and eligibility requirements.
  2. According to a Treasury press release, as of March 2023, 49% of HAF assistance was delivered to very low-income homeowners, defined as homeowners earning less than 50% of the area median income.
  3. See here for a letter that increased takeup of SSI benefits, here for a letter that did not increase flu vaccine uptake, and here for a letter that increased requests for wildfire risk assessments.
  4. University of Maryland Institutional Review Board (project reference number: 2006965-1) deemed the study exempt from full board review on January 20, 2023.
  5. A simple randomization was conducted, whereby the list of homeowners were assigned a uniformly distributed random number between 0-1, and the numbers were sorted in ascending order. The top half of the list was assigned to the holdout group, and the bottom half was assigned to the early mailer group.
  6. Unless noted otherwise, all of the analysis reported in this abstract was prespecified in an analysis plan.
  7. The final sample size was N = 1638 (n = 818 in the treatment group, n = 820 in the control group). A small group of homeowners who had applied prior to randomization were unintentionally included in the initial sample (this was not accounted for in our analysis plan). As a result, we removed these from our analysis.
  8. As per the analysis plan, the differences were adjusted for covariates. P-values were also adjusted to reflect that we conducted multiple statistical tests. P-values were calculated through randomization inference (RI) with 5000 replications. The results from the pre-registered RI tests yield the same substantive conclusions: a RI p-value of 0.334 for the treatment effect on “application initiated” and a RI p-value of 0.565 for the treatment effect on “application completed”.
  9. We tested each of a series of alternative null hypotheses ranging from -5% to +5%, in increments of 0.005%. This estimate is the smallest positive effect that we can reject with 95% confidence.
  10. Funding status is contingent on the applicant having submitted their application. Since application rates were very low for our sample, it would not have been possible to analyze this estimate, even if we had received this data.