Sign that says we welcome SNAP benefits

Sign that says we welcome SNAP benefits


View the PDF Version

Key findings

We sent proactive email communication addressing common application barriers to business owners who had initiated applications to a business development program. We found that this outreach did not increase business owners’ likelihood of submitting applications. However, this may be due to the short timeframe between the intervention and measurement of outcome data, during which very few initiated applications were submitted overall.

Agency priority

Spending over $650 billion per year, the U.S. government is the largest buyer of goods and services in the world. Executive Order 14091 includes a goal to increase the share of federal procurement money going to small disadvantaged businesses to 15% by fiscal year 2025. One way to meet this goal is to help prepare such businesses to participate in the federal marketplace through contracting assistance programs implemented by the U.S. Small Business Administration (SBA). The 8(a) Business Development Program (8(a) program) is a 9-year program that serves socially and economically disadvantaged small business owners. It provides participants with training and technical assistance to enhance their ability to compete in competitive business environments. Additionally, participating businesses are eligible for limited competition and noncompetitive (i.e., sole source) contracting opportunities with federal agencies. The SBA aims to increase participation in its 8(a) program to support the economic development of socially disadvantaged entrepreneurs and to promote equity in federal procurement.

Program change description

The 8(a) program application process involves numerous steps, including registering in federal contracting databases, filling out an application questionnaire, and uploading supporting documentation related to the business and its qualifying owner(s). Many business owners initiate applications to the 8(a) program but never submit them. Moreover, despite the abundant information and resources available on the application platform, the majority of submitted applications are incomplete.

To further increase the proportion of complete application submissions, we worked with the SBA to design new email outreach to individuals who had either initiated applications or submitted incomplete applications. The SBA had not previously conducted outreach to this population, and they were able to do so at no additional cost.

We designed two types of email outreach to address different potential barriers in the application process. To address the problem of incomplete applications, we compiled a checklist of application requirements to highlight those that are most often missing and to help firms assess their eligibility. To address complexities in the application process, we designed outreach to encourage applicants to meet with local partner organizations who can provide hands-on support for the 8(a) program application.1 In these emails, we also included contact information for the applicant’s nearest local partner organization.

Evaluation design

The evaluation included new applications initiated by businesses between April 15th and May 29th, 2024, as well as any applications initiated prior to April 15th but never submitted (N=15,811 applications).2 Outcome data recording application submissions were accessed on July 8th, 2024.3

Applications were randomly assigned to one of four groups.4 In total: 3,597 businesses served as our control group and did not receive an email; 3,684 received an email with the checklist of application requirements and criteria; 3,699 received an email encouraging them to meet with a local partner organization to fast-track their application; and 4,010 received an email containing both the checklist and the encouragement to meet with a local partner organization. The assignment was randomized at the application level, blocking on the order of application submission.

Analysis of existing data

This evaluation used data from Certify.sba.gov, the online portal businesses used to submit applications to the 8(a) program during the evaluation period.5 We analyzed the effects of the interventions on whether each application was submitted, whether it was screened as complete, and whether the applicant showed documentation of having met with a local partner organization.6

Results

Out of 15,811 initiated applications in our sample, 596 were submitted, 298 were screened as complete, and 50 included documentation of having met with a local partner organization. We estimated the effects of the checklist and local partner organization emails on whether applications were submitted, and the estimates were not statistically significant (p=.894 and p=.748).7 We also estimated the effects of both emails on whether applications were screened as complete and whether applicants met with a local partner organization, and the estimates were not statistically significant. Finally, we estimated the effects of receiving any email (comparing all treatment emails to the “no email” group) for all of our outcomes, and the estimates were not statistically significant.

Figure 1. Proactive email outreach did not increase the proportion of submitted applications8 Bar chart showing that proactive email outreach did not increase the proportion of submitted applications

Implications

The results of this evaluation suggest that proactive email outreach did not increase the number of submitted applications to the 8(a) program or the number of applications screened as complete over a ten-week period. The primary limitation of this evaluation was the relatively short time frame between the intervention and measurement of outcome data.9 Only a small fraction (4%) of businesses who had initiated applications submitted them during the evaluation period, meaning that we may have observed more submitted applications if we had received outcome data at a later date. A future follow-up study could potentially evaluate the outcomes of this evaluation using data collected over a longer time period to see if additional firms submit applications when given more time. If additional analyses with more data provide further evidence that the email outreach we designed did not increase application submission, then it may be useful to evaluate other strategies to make this type of outreach more effective.

Finally, while not a statistically significant finding, we observed in an exploratory analysis (not pre-registered) that applicants who received the local partner organization email were marginally more likely to have met with local partner organizations compared to those who did not receive this email.10 Future analyses could help us assess the robustness of this finding and design additional outreach to promote local partner organization meetings and application submissions to the 8(a) program.

Notes:

  1. The SBA provides counseling and training for small businesses and aspiring entrepreneurs through a variety of partner organizations, referred to here as “local partner organizations.” In our evaluation, we specifically directed applicants to their nearest APEX Accelerator. Under current program policy, applicants who show proof of meeting with a local partner organization can have their applications prioritized.
  2. Entity-owned businesses have a different set of rules but apply to the 8(a) program through the same system. We did not know which businesses were entity-owned at the time of submission, so we included them in the sample, even though we expected they would be less influenced by these interventions.
  3. We expected to receive outcome data on July 1st as noted in our analysis plan, but we actually received data on July 8th.
  4. Some businesses submitted multiple applications (0.4%) and others were inadvertently included in multiple rounds of randomization (4.8%), which means that they could have received multiple emails. We accounted for this in the analysis by defining intervention exposure at the business level and using inverse propensity and sample weights to correct for the differential probabilities of assignment.
  5. As of September 2024, Certify is the application legacy system, and SBA has begun accepting new applications through the MySBA portal.
  6. Unless otherwise noted, all of the analyses reported in this abstract were pre-specified in our analysis plan.
  7. P-values were calculated through randomization inference with 10,000 replications.
  8. For ease of presentation, we omitted the combined checklist and local partner organization group from this plot. We estimated the effect of each treatment on application submission and plotted the predicted mean for each group. The “no email” bar represents the weighted mean in the business-as-usual group. Error bars represent 95% confidence intervals.
  9. We discussed this issue in our analysis plan; see page 11.
  10. We ran exploratory analyses (not pre-registered) in which we omitted the interaction term from each model to estimate the average effect of each email. We found a marginal estimate of the effect of the local partner organization email on the outcome of meeting with a local partner organization (p=.057) and no other significant estimates.