Increasing procurement efficiency by automating market research

Summary

The goal of this evaluation was to test whether automating market research, a primary bottleneck in the federal procurement process, increased federal contracts for small businesses. We found that automating a key part of the market research process did not increase the probability of a contract to an SBA-certified small business, nor did it lead to a spillover effect on future contracts within the evaluation period.

Agency priority

Federal contracts accounted for over $750 billion in government spending in FY24. It is important that these funds are spent efficiently by optimizing the procurement process as well as awarding federal contracts to the best-suited businesses.

One of the primary bottlenecks in the federal procurement process is the market research required of contracting officers by the Federal Acquisition Regulation (FAR). Such market research is time consuming and varies widely. In addition, thorough market research can be inefficient and overwhelming due to the sheer volume of datasets available on federal contractors. The U.S. Small Business Administration (SBA) is responsible for supporting small businesses in obtaining federal contractsobtaining federal contracts. However, small businesses can be overlooked as potential contractors, including those certified in one of the SBA contracting assistance programs intended to strengthen the small business supplier base.

What we evaluated

We designed an intervention that automated the market research process and was tailored to the requirements of opportunities posted by contracting officers.¹ We compiled a list of firms for each combination of industry (NAICS code) and good/service (PSC code).² The lists were populated with basic information about each firm, including a link to their vendor page on SBA’s Dynamic Small Business Search tool.

The effectiveness of the automated market research intervention was evaluated with an individual-level randomized control trial among contracting officers who recently posted a contracting opportunity on SAM.gov. Some officers received no communication (“No email” condition). Others were simply encouraged to consider awarding the opportunity to an SBA-certified small business, without market research (“Encouragement” condition). A third group of contracting officers received the encouragement email along with a link to the automated market research list of firms in the opportunity’s NAICS/PSC combination (“AMR” condition). Over the course of the intervention period (July 2024 to January 2025), 830 “Encouragement” emails, 3,086 “AMR” emails were sent, and 4,807 opportunities received no email.

We analyzed the impact of the email outreach using data on federal procurement opportunities (SAM.gov) and contracts (Federal Procurement Data System, FPDS). For each contracting opportunity in our sample, we examined whether the ultimate awardee was an SBA-certified small business. This enabled us to examine effects at the opportunity level, though we were only able to match about 50% of opportunities to their subsequent awards or later opportunity postings.³ In addition, we estimated the indirect effect of the emails on office-level procurement behavior that is easier to measure.

What we learned

We did not find evidence that automating market research and emailing it to contracting officers significantly increased the proportion of contracts to SBA-certified small businesses. The Encouragement email also did not significantly affect contract patterns.

The percentage of contracts going to SBA-certified small businesses was 4.9% for opportunities among contracting officers receiving no communication, 3.9% for opportunities included in the “Encouragement” emails, and 6.5% for opportunities included in the “AMR” emails. This implies a decrease of 0.9 percentage points in the Encouragement group (p = 0.62, CI [-0.04, 0.03]) and an increase of 1.6 percentage points in the AMR group (p = 0.16, CI [-0.01, 0.04]), though the differences across the groups are not statistically significant.⁴

We found some evidence from one exploratory test that an office assigned to receive the AMR email, instead of the Encouragement email, obligated more funding to SBA-certified small businesses, though this result is not statistically significant.

Figure 1. Automating market research did not significantly increase the proportion of contracts to SBA-certified small businesses

Note: *p < 0.05

Additionally, a descriptive analysis of engagement with the market research web pages revealed that 4.52% of the market research links were viewed by those we sent them to, and that many people also visited the webpages through searches independently of the links we sent.⁵ This suggests that there is demand for market research tools. Further, analysis of the engagement data from web analytics provides suggestive evidence that there may be a correlation between engagement with the market research links and awards to SBA-certified small businesses.

Finally, another descriptive finding indicated that outreach early in the procurement process may be important. Opportunities entering our study at the “sources sought” stage were more likely to go to SBA-certified small businesses than opportunities entering our study at the “pre-solicitation” stage.

Applying the findings

There were several noteworthy limitations in this evaluation. First, we believe that the email outreach occurred after contracting officers had already made critical decisions about whether to contract with a small business. Future evaluations could examine whether supporting contracting officers with market research even earlier in the procurement process could maximize the gains from automation.

Second, we were able to match fewer opportunities to their subsequent contracts than we anticipated (50% instead of the anticipated 60-80%), and the intervention period was shorter than we planned (12 email cycles rather than 18); two challenges that reduced our statistical power. However, while we can rule out large positive effects we cannot rule out that there are moderate-sized positive effects from automating market research. Future evaluations could further assess the precise impact of automation on federal procurement and small businesses.

Notes:

1. We sampled opportunities at the sources sought or pre-solicitation stages that were not yet designated for SBA-certified small businesses. Opportunities were only sampled once, even if posted multiple times during our evaluation.
2. North American Industry Classification System (NAICS) and Product or Service Code (PSC).
3. We anticipated that we would only be able to match a subset of opportunities to awards due to limited linking variables across the two data sets, and followed our pre-registered analysis plan to account for this.
4. As pre-specified in the analysis plan, we conduct randomization inference for our primary hypotheses tests. One of these is a test of the difference in effects across the two treatment groups. The RI p-value for this test is 0.739, indicating the two treatment effects are not statistically significantly different.
5. Marketing companies suggest that effective email marketing campaigns might expect click-through rates (the rate at which recipients click on a link in an email) of about 2-5%.

Verify the upload date of our analysis plan on Github.