Evaluations

Identifying Interventions to Increase Affordable Care Act Uptake through a Systematic Review and Meta-analysis

Analysis Plan Registration

Does Reducing Documentation Burden Broaden Access to Emergency Rental Assistance? Quasi-experimental Evidence from Virginia

Analysis Plan Registration

Learning about Transparent Defaults for Opt Out Text Messages

Analysis Plan Registration

Impact Evaluation of Diversity, Equity, Inclusion, and Accessibility (DEIA) Training Program

Analysis Plan Registration

Increasing Naturalization Rates for Lawful Permanent Residents

Analysis Plan Registration

Increasing Applications to the Homeowner Assistance Fund Using Mailers - Pilot

Analysis Plan Registration

Measuring the Downstream Health Effects of SSI Take-up Among Older Adults

Analysis Plan Registration

Describing the Allocation of COVID-19 Relief Funds to Financial Institutions

Relief funds were awarded to financial institutions that serve low- and moderate-income and minority communities.

Investigating the Scope and Implications of Return to Title IV

Analysis Plan Registration

Incorporating Evaluation into Digital Forms

Five GSA offices and the American public worked together to improve federal forms

Equity in the Distribution of the Emergency Rental Assistance Program

A descriptive study of equity in the first-ever nationwide eviction prevention program

Understanding and Improving Tribal Governments’ and Non-Entitlement Units’ Experiences with the State and Local Fiscal Recovery Funds Program

Increasing Timely Reporting by Local Governments through Reminders

Sending reminder emails did not increase submission of spending reports

Increasing Timely Reporting by Local Governments through Peer Comparisons and Encouragements

Adding peer comparisons and information that registering is quick and easy to emails did not increase submission of spending reports

Increasing Timely Reporting by Local Governments through To-Do Lists

Adding to-do lists and calls to action to emails increased signup on an online expenditure reporting portal

Describing Candidate Assessments and Hiring Outcomes at DOI

Exploring means and proportions of recruitment success, time-to-hire, and candidate characteristics for hiring efforts at the Department of the Interior.

Survey Uptake Decisions with Transparent Default Choices

Disclosing the purpose of a default selection increases the likelihood that survey respondents choose to receive future contacts

Increasing Voluntary Tax Compliance for Return Preparers

A pre-filing season letter sent to preparers improved tax compliance

Reducing Inappropriate Prescribing and Co-Prescribing Behaviors - A Retrospective Analysis

A retrospective analysis of targeted letters to outlier prescribers of opioids and benzodiazepines

Increasing Take-Up of the American Opportunity Tax Credit

A series of emails and a letter increased take-up

Enhancing the Effect of Cash Buyback on Return of Unused Opioids

Reminder cards increased the return of unused prescription opioids

Evaluation Design for the Homeowner Assistance Fund

Working with the Department of the Treasury to design evaluations of a housing assistance program

Evaluation Design for the Emergency Rental Assistance Program

Working with the Department of the Treasury to design evaluations of a rental assistance program.

Evaluation Design for the State and Local Fiscal Recovery Fund

Working with the Department of the Treasury to design evaluations of a fiscal transfer program.

How Documentation Burdens Affect the Equitable Distribution of Small Business Relief Funding

A city small business relief program that reduced documentation requirements subsequently received a higher share of completed applications from historically underserved businesses

Who Receives Access to Small Business Relief? A Simulation-based Approach

Simulations using application data from three cities showed how outcomes for historically underserved businesses depend on the allocation method.

Evaluating the Effects of HEERF Aid to Students

A regression-discontinuity design to estimate the effects of emergency aid on student academic outcomes.

Understanding and Improving How Policymakers Respond to Program Impact

Decision aids can help policymakers incorporate evidence in program funding decisions.

Supporting NOAA’s Equity Assessment Teams

Evaluation options to build evidence for equity action plans at NOAA.

Dallas Small Business Continuity Fund Impact Evaluation

Analysis of a pre-existing lottery for small business grants and loans was unable to determine impact on business resilience

Increasing Voluntary Tax Compliance Through Outreach to Clients of Return Preparers

Letters sent to clients reduced likely errors in tax credit claims.

Evaluation of San Diego Small Business Relief Fund

A selection on observables analysis was unable to determine the effect of funding for small businesses on business outcomes during the pandemic.

Increasing Voluntary Contributions at Congregate Meals Program Sites

Using behavioral science to increase voluntary contributions in a senior meal program

Increasing Vaccine Uptake Through Performance Feedback

Provider vaccine performance feedback did not increase flu vaccine uptake among Veterans at the Atlanta VA

Increasing Requests for Wildfire Risk Assessments

Letters mailed to property owners increased requests for on-site wildfire risk assessments in Montana.

Encouraging Active Plan Choice Among LIS Beneficiaries

Revising letters to encourage Low-Income Subsidy (LIS) beneficiaries to make an active choice between Medicare Part D prescription drug plans.

Increasing Flu Vaccine Uptake Among Veterans at the Dorn VA Medical Center via a Reminder Postcard

Using targeted outreach to increase flu vaccine uptake among Veterans at the Dorn VA Medical Center

Increasing Utilization of Family Planning Services in Mozambique Through an SMS Intervention

Text reminders increased clinic visits for family planning services

Increasing Enrollment in Low-Income Medicare Programs

Revising letters to individuals who may be eligible for the Medicare Part D Low-Income Subsidy and Medicare Savings Program

Project SOAR

Education navigation program did not change FAFSA completion.

Reducing Concurrent Opioid-Benzodiazepine Prescriptions Through Provider Messages

Targeted messages to providers did not reduce concurrent prescriptions

Improving Claiming Choices for Recipients of Social Security Spousal Benefits

Revising a notice to increase filings for higher benefits.

Encouraging Social Security Income Recipients to Self-Report Wage Changes

A reminder encouraged SSI recipients to self-report changes in earnings earlier, potentially reducing overpayment.

Increasing Participation in Ticket to Work

Analysis Plan Registration

Characterizing Nonresponse Bias in the American Housing Survey (AHS)

Analysis Plan Registration

Reducing Nonresponse Bias in the American Housing Survey (AHS)

Analysis Plan Registration

Using Proactive Communication to Increase College Enrollment for Post-9/11 GI Bill Beneficiaries

A general proactive email, encouraging increased postsecondary enrollment, influenced the enrollment intensity of Post-9/11 GI Bill users

Increasing School Attendance in Seattle

Proactive communications do not change attendance outcomes

Using Social Norms to Decrease Energy Use in Public Housing

Letters with feedback on energy use and comparisons to efficient neighbors did not reduce energy use among public housing residents

Increasing Vaccine Uptake Among Veterans at the Atlanta VA Health Care System

A modified clinical reminder did not increase patient vaccine uptake.

Increasing Survey Response to Improve USAGov Email Content

A timeline making government operations transparent increased survey response and an early message of gratitude decreased survey response.

Increasing Use of Patient Generated Health Data through Patient Reminders

Sending reminder messages focused on provider accountability increased patients’ use of electronic blood glucose flow sheets.

Increasing Use of Patient Generated Health Data through Provider Encouragement

A virtual provider training and encouragement increased use of electronic blood glucose flow sheets.

Increasing FAFSA Completion Among Public Housing Residents: Seattle and King County

A pair of communications do not change FAFSA completion rates

Increasing FAFSA Completion Among Public Housing Residents: NYCHA

A series of communications do not change FAFSA completion rates

Improving Employment Services for UI Claimants in Oregon

Requiring personal employment plans did not change the employment rate

Communicating Employment Supports to Denied Disability Insurance Applicants

Letters to denied disability applicants about employment support programs did not reduce appeals.

Integrating Health Care Services in Nigeria

A feasibility pilot to increase access to voluntary family planning services.

Email sent via GovDelivery to Veterans enrolled with the VA New York Harbor

Emails incorporating behavioral insights were not effective in increasing flu shot uptake.

Decreasing Energy Costs in Federally Assisted Housing

Sending flyers with energy-saving tips did not change energy use.

Increasing Flu Vaccine Uptake Among Veterans at the St. Cloud VA

Postcards incorporating behavioral insights were not more effective than a basic informational postcard.

Increasing Vaccine Uptake Among Seniors in Louisiana

Sending postcard reminders increases vaccination rates among elderly

Increasing Immunization Compliance in Schools

Data visualization for school-specific immunization compliance reports did not impact compliance rates.

Increasing Flu Shot Uptake Among Pregnant Women

Targeted email outreach via an electronic medical record messaging system did not increase uptake of the flu vaccine among pregnant women.

Increasing Flexible Spending Account (FSA) Participation Among Government Employees

Targeted email outreach to GSA employees did not increase enrollment in Flexible Spending Accounts.

Increasing AHS Response Rates

A second advance letter does not significantly improve outcomes.

Increasing SSI Uptake Among a Potentially Eligible Population

Letters to adults 65 or older increased SSI awards by 340%.

Increasing ART Adherence and Retention in Care for HIV Positive Women in Ethiopia

Phone calls delivered to HIV+ women did not increase ART adherence.

Preventing Malaria During Pregnancy

Redesigned referral process did not increase antenatal care (ANC) attendance for pregnant women.

Identifying Students for the Education for Homeless Children and Youth Program

Identifying Homeless Students for Educational Support Services

Increasing Awareness of Health Insurance Options

Letters about tax penalties can increase health insurance coverage

Increasing Pre- and Post-Natal Care Uptake Among Medicaid-Eligible Mothers

Home Visiting Uptake Among Medicaid-Eligible Mothers.

Reducing Overprescribing of Quetiapine in Medicare Part D

Peer comparison letters for high prescribers of quetiapine reduce prescription volume and improve guideline conformity of prescription fills.

Matching Government Surplus Property with Interested Buyers

Customized emails reduced no-bids and increased overall revenue on gsaauctions.gov.

Testing Variations of Maternal Immunization Messages

Variations of an ad did not lead to detectable differences in site visits.

Increasing Veteran Health Care Benefits Enrollment

Emails to transitioning servicemembers about VA health care benefits did not affect enrollment.

Increasing TSP Enrollment among USPS Employees

Testing mailings to increase USPS retirement enrollment and contributions

Increasing FAFSA Renewal Rates

Reminder emails sent to current students led to an increase of approximately 250,000 FAFSA submissions.

Encouraging Action Among Students At Risk to Withdraw

Reminding students to reapply for financial aid or begin loan repayment decreased postsecondary enrollment by one percentage point.

Increasing Voluntary Submission of Demographics Information with USAJOBS Applications

The combination of default settings and simplified language increases voluntary submission of demographics information by 19%.

Prompting Decisions on Retirement Saving

Encouraging higher TSP contribution rates during new employee orientation showed no detectable effect on the likelihood of contributing 5% or more.

Improving Capacity of Federal Managers through Project Management

A 16-week training course for Federal Supervisors shows potential to improve organizational culture and performance.

Encouraging Microloan Applications from Women-Owned Farms

A single direct mailer had no statistically reliable effect on the likelihood that female farmers who received it would apply for microloans.

Increasing Military Patient Use of Secure Messaging through Email Reminders

Emails sent after an appointment-based intervention had no statistically reliable impact on patient registration for secure messaging.

Increasing Military Patient Use of Secure Messaging through Personal Appeal and Assistance

Provider encouragement and on-the-spot assistance increased patient registrations for secure messaging.

National School Lunch Program Eligibility Verification

Timely communication with beneficiaries has mixed effects on response rates.

Industrial Funding Fee Reports Update

A confirmation prompt reduced financial self-reporting errors initially, but the effect did not persist in subsequent periods.

Simple Suggestion Increases Savings Enrollment, Contributions

Suggesting contribution rates increases savings enrollment and contributions

Increasing TSP Enrollment Rates through Active Choice

Presenting TSP enrollment as an active choice during in-processing increased enrollment by 6-7%.

Increasing Newsletter Subscription Rates of Active Duty Servicemembers

Clear communications emphasizing active choice increase subscription rates.

Encouraging Engagement of Military Caregivers

Emails to caregivers of ill or wounded servicemembers increased awareness of available benefits.

Increasing FAFSA Completion by HUD-Assisted Youth

Sending letters to HUD-assisted youth had no significant effect on FAFSA completion rates.

Increasing Enrollment in the Small Business Administration's Learning Center

Simplifying a registration form increased learning course completion and data collection.

Improving Voter Participation in Local USDA Elections

Sending redesigned ballot materials to eligible farmers increased voter participation by 2.9%.

Decreasing Non-responses to NSLP Income Verification Requests

Simplifying the income verification letter for National School Lunch Program recipients increased response rate by 2.1%.

Increasing Tax Filing Among Lower-Income Individuals

Sending postcards effectively increased uptake of Earned Income Tax Credit among eligible population.

Increasing Health Insurance Enrollment Among Uninsured Americans

Modified emails encourage open rates about health insurance enrollment.

Encouraging Medicare Beneficiaries to Self-Identify as ACO Patients

Sending a letter to Medicare beneficiaries increased voluntary alignment to Accountable Care Organizations.

Increasing Patient Use of Health IT

Revising patient instructions following appointments led to 9.98% increase in online health portal activation.

Increasing Student Loan Rehabilitation Rates for Defaulted Borrowers

Emphasizing the negative consequences of loan default increases entry into loan rehabilitation programs.

Increasing IDR Re-certification Among Student Borrowers

Including required payment amount increased IDR recertification by 8.4%.

Increasing IDR Applications Among Eligible Student Loan Borrowers - Scale Up

Email scale-up resulted in more than 6,000 borrowers signing up for income driven repayment plans.

Increasing IDR Applications Among Eligible Student Loan Borrowers through Targeted Messages

Emails increase application submissions for income driven repayment plans.

Increasing Access to PSLF Benefits Among Eligible Population

Emails increased Public Service Loan Forgiveness form submissions three-fold without increasing the error rate.

Increasing Military Retiree Reactivation of Online MyPay Accounts

Emails increased myPay account reactivation by military retirees.

Encouraging Flu Vaccine Uptake among Medicare Beneficiaries

Letters sent to Medicare beneficiaries increased uptake of the flu vaccine.

Reducing Summer Melt

Eight personalized text messages to low-income students resulted in a 6% increase in college enrollment.

Reducing Self-reporting Errors by IFF Form Users

Adding the signature box resulted in an extra $1.59 million paid to government by vendors.

Increasing Post-Grace Period Loan Payments by Student Borrowers

Emails sent to student loan borrowers at risk of default increased payments.

Increasing Satisfaction Survey Response Rate from Federal Tenants - Time of Day

Emails sent at lunchtime saw 1.5% more Federal employees click through to a workplace survey.

Increasing Satisfaction Survey Response Rate from Federal Tenants - Subject Lines and Day of Week

Emails with subject lines that included the word "please" increased Federal employees responses to a workplace survey.

Increasing Satisfaction Survey Response Rate from Federal Tenants - Calendar Invite

Emails including a calendar invite reminder led to increased survey responses from Federal employees.

Increasing Chapter 36 Benefits Uptake by Veterans

Emails informing veterans about education and career benefits increased applications.

Reducing Inappropriate Prescribing of Schedule II Substances in Medicare Part D

Sending letters to potential overprescribers did not influence Schedule II prescribing activities.

Encouraging Account Transfers by Securities Holders

Scheduling appointments to transfer securities increased call-in rates by 2.37%.

Increasing Farmer Microloan Applications

A new letter increases farmers' access to credit by 22%.

Increasing Online TSP Enrollment through Emails

A single email to military Servicemembers nearly doubled the rate of savings plan enrollment.

Increasing Servicemember On-Base TSP Enrollment

Explaining benefits of TSP during in-processing doubled the number of Servicemembers enrolling.

Increasing Duplex Printing Among Federal Workforce

A digital pop up significantly lowers office-wide printing costs.

Increasing Health Insurance Enrollment Among Uninsured Americans

A letter increased health insurance enrollment by 7.15%.

Increasing IDR Applications Among Delinquent Student Borrowers

Sending emails to delinquent borrowers positively impacted IDR application completion.

Increasing Servicemember Roth TSP Re-Enrollment

One email enables 22% more Servicemembers to re-enroll in their Thrift Savings Plans.

Increasing Debt Repayment by Delinquent Lenders

A modified letter sent to delinquent borrowers increased utilization of online payments.

Understanding and Improving Tribal Governments’ and Non-Entitlement Units of Local Governments’ Experience With the State and Local Fiscal Recovery Funds Program

Enhancing the Effect of Cash Buyback on Return of Unused Opioid Pills

Reminder cards increased the return of unused prescription opioids