Dallas small business continuity fund impact evaluation

What was the challenge?

The Small Business Administration (SBA) aims to restore small business revenue and sustain jobs after disasters. SBA is interested in understanding the impact of community-based approaches to help small businesses respond and recover to the COVID-19 pandemic. In spring 2020, the Dallas City Council used CARES Act funding to set up the Small Business Continuity Fund (SBCF) to help small businesses affected by the economic fallout from the COVID‐19 pandemic. We partnered with the City of Dallas to understand how receiving a grant or a loan affected businesses’ ability to generate revenue and remain open, as well as the differential impact of such funding for women-owned and black-and-hispanic-owned businesses.

What was the program change?

We worked with the City of Dallas to understand the lottery implementation in detail. This understanding allowed us to approximate the conditions of a randomized controlled trial by reconstructing the assignment probabilities implied by the lottery procedure.

The grant and loan lotteries randomly assigned a rank to each business application that passed initial eligibility checks. That rank determined the order in which the City of Dallas sent businesses invitations to follow up with additional documentation for funding. By simulating the lottery process 10,000 times, we calculated each business’s probability of being invited to submit documentation for funding on a specific day in 2020. These probabilities are illustrated in Figure 1.

Figure 1. Daily probability of receiving an invitation to submit documents for funding by lottery
Note: Solid green line is for grants in non-targeted areas, dotted blue line is for grants in targeted areas, dashed red line is for loans.

How did the evaluation work?

Figure 1 shows some applications have a higher daily probability of invitation than others. We used inverse propensity weights to correct for this skew in the group composition over time, which might have otherwise biased the estimation. We first estimated the effect of being invited to submit documents for funding. This estimate is then scaled up by a factor proportional to the rate at which invited businesses were funded, using a procedure called two-stage instrumental variables regression, to estimate the effect of receiving funding among those businesses that would receive funding if they were invited.

To assess how funding affected business resilience during the pandemic, we matched the lottery data with a panel of outcome data for every day in 2020, including federal bankruptcy court data and data from a business rating review platform company on whether businesses were open or closed on a given day and whether they offered virtual services (delivery, online classes and consultations, etc.).

What was the impact?

We did not find statistically significant evidence of program impact across any of the main analyses specified in the analysis plan: business closures, provision of online services, and bankruptcy filings (see Figure 2). This should not be mistaken with finding evidence that the program did not work: given the data, we cannot rule out either positive or negative program impacts. The inability to say more relates to the statistical uncertainty in the estimates. The large number of applications suggests business owners saw a great need for the funding.

Figure 2. The estimated effect of funding on business closures, online services, and bankruptcy

Note: Blue circles show estimated effect of invitation, yellow triangles show estimated effect of funding, among businesses that would receive funding if invited.

Key takeaways

We identified at least two directions for future work to improve both program implementation and evaluation. First, building comprehensive and easily accessible datasets on the small business population. Access to EIN, address, and quarterly wage bill of all of the business establishments in a jurisdiction, could allow outreach to be better targeted and support additional research on employment impacts of relief programs. Second, prioritizing additional evaluation activities on how to increase follow-up by relief applicants. Most businesses (56%) who were invited to submit documents for funding simply did not follow up or withdrew. Reducing this rate might improve the efficiency of relief fund distribution and enhance ability to measure its impact.

Verify the upload date of our Analysis Plan on GitHub.