Evaluations

Evaluating Artificial Intelligence users needs

Learning about employees’ experience and support needs while using AI

Reducing filing errors via education and outreach to tax preparers

Analysis plan registration

Reducing filing errors via outreach to tax preparers and clients

Analysis plan registration

Understanding rural small businesses that successfully win federal contracts

Analysis Plan Registration

Boost the Made in America Manufacturing Initiative by identifying bottlenecks in the small manufacturer pipeline

Analysis Plan Registration

Requiring electronic manifests of hazardous waste generated by EPA-led cleanup sites

Analysis Plan Registration

Increasing Medicaid renewals with text message reminders

Additional messages and phone calls aimed at improving credibility of government communications did not affect the rate of health coverage renewals

Reducing no-shows for in-person passport appointments

Sending automated text message reminders to customers did not change the no-show rate for in-person passport appointments

Descriptive study of visitation trends at FEMA Disaster Recovery Centers

Analysis Plan Registration

Evaluating the effect of changes to the Low Income Home Energy Assistance Program

Analysis plan registration

Increasing applications for contracting assistance among underserved small businesses through proactive outreach

Sending emails to business owners who had initiated applications to the 8(a) business development program did not increase their likelihood of submitting those applications

Examining the impacts of COVID-era direct payments during the transition to adulthood

Analysis plan registration

Decreasing Supplemental Nutrition Assistance Program (SNAP) denials due to incomplete information

A text campaign encouraging the use of a redesigned document uploader decreased SNAP denial rates due to incomplete eligibility information

Decreasing abandonment of calls to the 988 Suicide & Crisis Lifeline

An updated caller experience resulted in more people being connected to a 988 counselor

Measuring the impact of the Appropriate Responses Initiative on arrests in Ramsey County, MN

Analysis Plan Registration

Modifying income reporting on benefits applications — Evidence from households with multiple jobs

Providing applicants with a more structured way to report their income did not increase submission rates for an online application for SNAP benefits

Modifying income reporting on benefits applicants — Evidence from single-income households

Providing applicants with different alternatives to reporting an hourly wage did not increase submission rates for an online application for SNAP benefits

Decreasing homelessness and financial instability with unconditional cash transfers

Analysis plan registration

Streamlining income verification to broaden access to rental assistance

Streamlining income verification for application processors in Kentucky increased application approval rates by at least 7 percentage points

Reducing documentation burdens to accelerate and broaden access to emergency rental assistance

Simplifying income documentation reduced application processing times and increased application approval rates, but did not significantly increase new applications

Increasing access to Minnesota Child Care Stabilization Base Grants

Texts and calls increased funding awards for child care providers

Decreasing opt out from text messages about benefit programs

Measuring delivery and opt out-rates for a text message campaign with a state partner

Encouraging the use of USCIS online accounts

Evaluating the impact of communications on online account creation and case linkage

Encouraging the use of USCIS’ online forms

Evaluating methods to increase online filing for Permanent Resident Card renewals

Identifying interventions to increase Affordable Care Act uptake through a systematic review and meta-analysis

Support-based interventions increase health insurance enrollment

Increasing applications to the Homeowner Assistance Fund using mailers

Sending postcards did not increase applications from at-risk homeowners

Describing the allocation of COVID-19 relief funds to financial institutions

Relief funds were awarded to financial institutions that serve low- and moderate-income and minority communities

Investigating the scope and implications of Return to Title IV funds

Building foundational descriptive evidence that documents the landscape and scope of R2T4

Improving operational processes at USCIS

Analysis Plan Registration

Describing candidate assessments and hiring outcomes at DOI

Exploring means and proportions of recruitment success, time-to-hire, and candidate characteristics for hiring efforts at the U.S. Department of the Interior

Increasing voluntary tax compliance through outreach to clients of return preparers

Letters sent to clients reduced likely errors in tax credit claims

Increasing participation in Ticket to Work through redesigned mailers

Redesigned mailings did not increase Ticket assignments, but had a small impact on Helpline calls

Using incentives to reduce nonresponse bias in the American Housing Survey

Targeting incentives increased the response rate but did not decrease nonresponse bias

Measuring the downstream health effects of SSI take-up among older adults

Analysis Plan Registration

Incorporating evaluation into digital forms

Five GSA offices and the American public worked together to improve federal forms

Describing the distribution of Emergency Rental Assistance funds to those who need and qualify for it

A descriptive evaluation of the first-ever nationwide eviction prevention program

Understanding and improving Tribal governments’ and non-entitlement units’ experiences with the State and Local Fiscal Recovery Funds program

A descriptive evaluation of the State and Local Fiscal Recovery Funds program

Increasing timely reporting by local governments through reminders

Sending reminder emails did not increase submission of spending reports

Increasing timely reporting by local governments through peer comparisons and encouragements

Adding peer comparisons and information that registering is quick and easy to emails did not increase submission of spending reports

Increasing timely reporting by local governments through to-do lists

Adding to-do lists and calls to action to emails increased signup on an online expenditure reporting portal

Survey uptake decisions with transparent default choices

Disclosing the purpose of a default selection increases the likelihood that survey respondents choose to receive future contacts

Increasing voluntary tax compliance for return preparers

A pre-filing season letter sent to preparers improved tax compliance

Reducing inappropriate prescribing and co-prescribing behaviors - A retrospective analysis

A retrospective analysis of targeted letters to outlier prescribers of opioids and benzodiazepines

Increasing take-up of the American Opportunity Tax Credit

A series of emails and a letter increased take-up

Enhancing the effect of cash buyback on the return of unused opioids

Reminder cards increased the return of unused prescription opioids

How documentation burdens affect the distribution of small business relief funding

A city small business relief program that reduced documentation requirements subsequently received a higher share of completed applications from historically underserved businesses

Evaluating the effects of HEERF Aid to students

A regression-discontinuity design to estimate the effects of emergency aid on student academic outcomes

Understanding and improving how policymakers respond to program impact

Decision aids can help policymakers incorporate evidence in program funding decisions

Dallas small business continuity fund impact evaluation

Analysis of a pre-existing lottery for small business grants and loans was unable to determine impact on business resilience

Evaluation of San Diego Small Business Relief Fund

A selection on observables analysis was unable to determine the effect of funding for small businesses on business outcomes during the pandemic

Increasing voluntary contributions at congregate meals program sites

Using behavioral science to increase voluntary contributions in a senior meal program

Increasing vaccine uptake through performance feedback

Provider vaccine performance feedback did not increase flu vaccine uptake among Veterans at the Atlanta VA

Increasing requests for wildfire risk assessments

Letters mailed to property owners increased requests for on-site wildfire risk assessments in Montana

Encouraging active plan choice among LIS beneficiaries

Revising letters to encourage Low-Income Subsidy (LIS) beneficiaries to make an active choice between Medicare Part D prescription drug plans

Increasing flu vaccine uptake among Veterans at the Dorn VA Medical Center via a reminder postcard

Using targeted outreach to increase flu vaccine uptake among Veterans at the Dorn VA Medical Center

Increasing utilization of family planning services in Mozambique through an SMS intervention

Text reminders increased clinic visits for family planning services

Increasing enrollment in low-income Medicare programs

Revising letters to individuals who may be eligible for the Medicare Part D Low-Income Subsidy and Medicare Savings Program

Project SOAR

Education navigation program did not change FAFSA completion

Reducing concurrent opioid-benzodiazepine prescriptions through provider messages

Targeted messages to providers did not reduce concurrent prescriptions

Improving claiming choices for recipients of Social Security spousal benefits

Revising a notice to increase filings for higher benefits

Encouraging social security income recipients to self-report wage changes

A reminder encouraged SSI recipients to self-report changes in earnings earlier, potentially reducing overpayment

Using proactive communication to increase college enrollment for post-9/11 GI Bill beneficiaries

A general proactive email encouraging increased postsecondary enrollment influenced the enrollment intensity of Post-9/11 GI Bill users

Increasing school attendance in Seattle

Proactive communications did not change attendance outcomes

Using social norms to decrease energy use in public housing

Letters with feedback on energy use and comparisons to efficient neighbors did not reduce energy use among public housing residents

Increasing vaccine uptake among veterans at the Atlanta VA Health Care System

A modified clinical reminder did not increase patient vaccine uptake

Increasing survey response to improve USAGov email content

A timeline making government operations transparent increased survey response and an early message of gratitude decreased survey response

Increasing use of patient generated health data through patient reminders

Sending reminder messages focused on provider accountability increased patients’ use of electronic blood glucose flow sheets

Increasing use of patient generated health data through provider encouragement

A virtual provider training and encouragement increased use of electronic blood glucose flow sheets

Increasing FAFSA Completion among public housing residents - Seattle and King County

A pair of communications did not change FAFSA completion rates

Increasing FAFSA completion among public housing residents - NYCHA

A series of communications did not change FAFSA completion rates

Improving employment services for UI claimants in Oregon

Requiring personal employment plans did not change the employment rate

Communicating employment supports to denied disability insurance applicants

Letters to denied disability applicants about employment support programs did not reduce appeals

Integrating health care services in Nigeria

A feasibility pilot to increase access to voluntary family planning services

Email sent via GovDelivery to Veterans enrolled with the VA New York Harbor

Emails incorporating evidence-based insights were not effective in increasing flu shot uptake

Decreasing energy costs in federally assisted housing

Sending flyers with energy-saving tips did not change energy use

Increasing flu vaccine uptake among Veterans at the St. Cloud VA

Postcards incorporating behavioral insights were not more effective than a basic informational postcard

Increasing vaccine uptake among seniors in Louisiana

Sending postcard reminders increased vaccination rates among elderly

Increasing immunization compliance in schools

Data visualization for school-specific immunization compliance reports did not impact compliance rates

Increasing flu shot uptake among pregnant women

Targeted email outreach via an electronic medical record messaging system did not increase uptake of the flu vaccine among pregnant women

Increasing Flexible Spending Account (FSA) participation among government employees

Targeted email outreach to GSA employees did not increase enrollment in Flexible Spending Accounts

Increasing AHS response rates

A second advance letter did not significantly improve outcomes

Increasing SSI uptake among a potentially eligible population

Letters to adults 65 or older increased SSI awards by 340%

Increasing ART adherence and retention in care for HIV positive women in Ethiopia

Phone calls delivered to HIV+ women did not increase ART adherence

Preventing malaria during pregnancy

Redesigned referral process did not increase antenatal care (ANC) attendance for pregnant women

Identifying students for the Education for Homeless Children and Youth Program

Identifying homeless students for educational support services

Increasing awareness of health insurance options

Letters about tax penalties can increase health insurance coverage

Increasing pre- and post-natal care uptake Among Medicaid-eligible mothers

Home visiting uptake among Medicaid-eligible mothers

Reducing overprescribing of Quetiapine in Medicare Part D

Peer comparison letters for high prescribers of quetiapine reduce prescription volume and improve guideline conformity of prescription fills

Matching government surplus property with interested buyers

Customized emails reduced no-bids and increased overall revenue on gsaauctions.gov

Evaluating variations of maternal immunization messages

Variations of an ad did not lead to detectable differences in site visits

Increasing Veteran health care benefits enrollment

Emails to transitioning servicemembers about VA health care benefits did not affect enrollment

Increasing TSP enrollment among USPS employees

Evaluating mailings to increase USPS retirement enrollment and contributions

Increasing FAFSA renewal rates

Reminder emails sent to current students led to an increase of approximately 250,000 FAFSA submissions

Encouraging action among students at risk to withdraw

Reminding students to reapply for financial aid or begin loan repayment decreased postsecondary enrollment by 1 percentage point

Increasing voluntary submission of demographic information with USAJOBS applications

The combination of default settings and simplified language increases voluntary submission of demographics information by 19%

Prompting decisions on retirement savings

Encouraging higher TSP contribution rates during new employee orientation showed no detectable effect on the likelihood of contributing 5% or more

Improving capacity of federal managers through project management

A 16-week training course for federal supervisors shows potential to improve organizational culture and performance

Encouraging microloan applications from women-owned farms

A single direct mailer had no statistically reliable effect on the likelihood that female farmers who received it would apply for microloans

Increasing military patient use of secure messaging through email reminders

Emails sent after an appointment-based intervention had no statistically reliable impact on patient registration for secure messaging

Increasing military patient use of secure messaging through personal appeal and assistance

Provider encouragement and on-the-spot assistance increased patient registrations for secure messaging

National school lunch program eligibility verification

Timely communication with beneficiaries has mixed effects on response rates

Industrial funding fee reports update

A confirmation prompt initially reduced financial self-reporting errors, but the effect did not persist in subsequent periods

Simple suggestion increases savings enrollment, contributions

Suggesting contribution rates increases savings enrollment and contributions

Increasing TSP enrollment rates through active choice

Presenting TSP enrollment as an active choice during in-processing increased enrollment by 6-7%

Increasing newsletter subscription rates of active duty Servicemembers

Clear communications emphasizing active choice increase subscription rates

Encouraging engagement of military caregivers

Emails to caregivers of ill or wounded servicemembers increased awareness of available benefits

Increasing FAFSAcompletion by HUD-assisted youth

Sending letters to HUD-assisted youth had no significant effect on FAFSA completion rates

Increasing enrollment in the Small Business Administration's Learning Center

Simplifying a registration form increased learning course completion and data collection

Improving voter participation in local USDA elections

Sending redesigned ballot materials to eligible farmers increased voter participation by 2.9%

Decreasing non-responses to NSLP income verification requests

Simplifying the income verification letter for National School Lunch Program recipients increased response rate by 2.1%

Increasing tax filing among lower-income individuals

Sending postcards effectively increased uptake of Earned Income Tax Credit among eligible population

Increasing health insurance enrollment among uninsured Americans

Modified emails encourage open rates about health insurance enrollment

Encouraging Medicare beneficiaries to self-identify as ACO patients

Sending a letter to Medicare beneficiaries increased voluntary alignment to Accountable Care Organizations

Increasing patient use of health IT

Revising patient instructions following appointments led to 9.98% increase in online health portal activation

Increasing student loan rehabilitation rates for defaulted borrowers

Emphasizing the negative consequences of loan default increases entry into loan rehabilitation programs

Increasing IDR re-certification among student borrowers

Including required payment amount increased IDR recertification by 8.4%

Increasing IDR applications among eligible student loan borrowers - scale up

Email scale-up resulted in more than 6,000 borrowers signing up for income driven repayment plans

Increasing IDR applications among eligible student loan borrowers through targeted messages

Emails increase application submissions for income driven repayment plans

Increasing access to PSLF Benefits among eligible population

Emails increased Public Service Loan Forgiveness form submissions three-fold without increasing the error rate

Increasing military retiree reactivation of online myPay accounts

Emails increased myPay account reactivation by military retirees

Encouraging flu vaccine uptake among Medicare beneficiaries

Letters sent to Medicare beneficiaries increased uptake of the flu vaccine

Reducing summer melt

Eight personalized text messages to low-income students resulted in a 6% increase in college enrollment

Reducing self-reporting errors by IFF form users

Adding the signature box resulted in an extra $1.59 million paid to government by vendors

Increasing post-grace period loan payments by student borrowers

Emails sent to student loan borrowers at risk of default increased payments

Increasing satisfaction survey response rate from federal tenants - time of day

Emails sent at lunchtime saw 1.5% more federal employees click through to a workplace survey

Increasing satisfaction survey response rate from federal tenants - subject lines and day of week

Emails with subject lines that included the word "please" increased federal employees responses to a workplace survey

Increasing satisfaction survey response rate from federal tenants - calendar invite

Emails including a calendar invite reminder led to increased survey responses from federal employees

Increasing Chapter 36 benefits uptake by Veterans

Emails informing veterans about education and career benefits increased applications

Reducing inappropriate prescribing of Schedule II substances in Medicare Part D

Sending letters to potential overprescribers did not influence Schedule II prescribing activities

Encouraging account transfers by securities holders

Scheduling appointments to transfer securities increased call-in rates by 2.37%

Increasing farmer microloan applications

A new letter increases farmers' access to credit by 22%

Increasing online TSP enrollment through emails

A single email to military Servicemembers nearly doubled the rate of savings plan enrollment

Increasing Servicemember on-base TSP enrollment

Explaining benefits of TSP during in-processing doubled the number of Servicemembers enrolling

Increasing duplex printing among the federal workforce

A digital pop-up significantly lowers office-wide printing costs

Increasing health insurance enrollment among uninsured Americans

A letter increased health insurance enrollment by 7.15%

Increasing IDR applications among delinquent student borrowers

Sending emails to delinquent borrowers positively impacted IDR application completion

Increasing Servicemember Roth TSP re-enrollment

One email enables 22% more Servicemembers to re-enroll in their Thrift Savings Plans

Increasing debt repayment by delinquent lenders

A modified letter sent to delinquent borrowers increased utilization of online payments

Increasing application submissions for the 8(a) business development program

Decreasing Supplemental Nutrition Assistance Program (SNAP) denials due to incomplete information

Decreasing abandonment of calls to the 988 Suicide & Crisis Lifeline

Modifying income reporting on benefits applications - Evidence from single-income households

Modifying income reporting on benefits applications - Evidence from single-income households

Streamlining income verification to broaden access to rental assistance

Identifying interventions to increase Affordable Care Act uptake through a systematic review and meta-analysis

Reducing documentation burdens to accelerate and broaden access to emergency rental assistance

Increasing access to Minnesota Child Care Stabilization Base Grants

Decreasing opt out from text messages about benefit programs

Encouraging the use of USCIS online accounts

Encouraging the use of USCIS online forms

Increasing applications to the Homeowner Assistance Fund using mailed outreach

Incorporating evaluation into digital forms

Understanding and improving Tribal governments’ and non-entitlement units of local governments’ experience With the State and Local Fiscal Recovery Funds program

Increasing participation in Ticket To Work through redesigned mailers

Using incentives to reduce nonresponse bias in the American Housing Survey

Enhancing the effect of cash buyback on the return of unused opioid pills

Reminder cards increased the return of unused prescription opioids