Below are selected publications resulting from OES collaborations.

All of our work is available on our website, regardless of outcome, in line with research best practices. Our first priority is to make project results accessible and actionable for agency partners. More information about publications resulting from OES collaborations can be found on our Google Scholar page.

2026

Toma, M., Burnett, R., Debroy, P., Dimant, E., Liu, J., Safran, E., Saya, U., & Schultz, B. (2026). Promoting safe opioid disposal: Experimental evidence on behavioral messaging with financial incentives. Health Economics, 1–10. https://doi/10.1002/hec.70085.

  • This field experiment tested whether behaviorally informed reminder cards increase participation in a financial incentive program for returning unused opioid pills among U.S. Veterans. Relative to a cash incentive alone, the reminder cards increase both the likelihood that patients return unused pills (the extensive margin) and the number of pills returned (the intensive margin).

2025

Hemmeter, J., Phillips, J., Safran, E., & Wilson, N. (2025). Communicating program eligibility: A Supplemental Security Income (SSI) field experiment. American Economic Journal: Economic Policy, 17(2): 446–70. https://doi.org/10.1257/pol.20210560.

  • We conducted a direct mail field experiment with 4,016,461 individuals to test several key hypotheses about why take-up of Supplemental Security Income among individuals age 65 and above is so low. Communicating likely eligibility in a basic letter generated substantial increases in take-up in relative terms. Yet, the application rate in our study sample during the full 24-month follow-up period remained no greater than 7%. Our results reveal a modest tradeoff between increasing applications and the conditional likelihood of award, as well as the award amount. Ungated access.

2024

Martin, L., Feher, A., Schultz, W., Safran, E., & Cohen, A. K. (2024). Interventions to increase Affordable Care Act Marketplace enrollment: A systematic review and meta-analysis. Population Health Management, 27(5), 327–337. https://doi.org/10.1089/pop.2024.0091.

  • This study summarizes published and unpublished evidence on the efficacy of behaviorally informed interventions designed to encourage enrollment in subsidized health insurance coverage through the Affordable Care Act (ACA) marketplaces, and suggests which administrative burdens might be most relevant for potential enrollees. The meta-analysis found that behaviorally informed interventions can increase ACA marketplace enrollment. Interventions aimed at alleviating compliance costs by providing enrollment support were about three times as effective as information alone.

2023

Debroy, P., Balu, R., Burnett, R., Johnson, R., Kappes, H. B., Wallace, J. M., & Marconi, V. C. (2023). A cluster randomized controlled trial of a modified vaccination clinical reminder for primary care providers. Health Psychology, 42(3), 195–204. https://doi.org/10.1037/hea0001218.

  • We tested a provider-focused vaccination uptake intervention: a modified electronic health record clinical reminder that bundled together three adult vaccination reminders, presented patient vaccination history, and included talking points for providers to address vaccine hesitancy. Conclusion: Provider-focused interventions are a promising way to address vaccine hesitancy, but they may need to be more intensive than a modified clinical reminder to have appreciable effects on vaccination uptake. Ungated access.

Kappes, H. B., Toma, M., Balu, R., Burnett, R., Chen, N., Johnson, R., Leight, J., Omer, S. B., Safran, E., Steffel, M., Trump, K., Yokum, D., & Debroy, P. (2023). Using communication to boost vaccination: Lessons for COVID-19 from evaluations of eight large-scale programs to promote routine vaccinations. Behavioral Science & Policy, 9(1), 11-24. https://doi.org/10.1177/23794607231192690.

  • The COVID-19 pandemic has added new urgency to the question of how best to motivate people to get needed vaccines. In this article, we present lessons gleaned from government evaluations of eight large randomized controlled trials of interventions that used direct communications to increase the uptake of routine vaccines. These evaluations, conducted by the U.S. General Services Administration’s Office of Evaluation Sciences (OES) before the start of the pandemic, had a median sample size of 55,000. Participating organizations deployed a variety of behaviorally informed direct communications and used administrative data to measure whether people who received the communications got vaccinated or took steps toward vaccination.

2022

Feher, A., Hopkins, D. J., Safran, E., Peck, J., & Yokum, D. (2022). Effectiveness of behaviorally informed letters on health insurance marketplace enrollment: A randomized clinical trial JAMA Health Forum, 3(3):e220034. https://doi.org/10.1001/jamahealthforum.2022.0034.

  • In this randomized clinical trial that included 744,510 individuals on the HealthCare.gov platform during the final 2 weeks of the 2015 open enrollment period, use of a single behaviorally informed letter caused a statistically significant increase in health insurance enrollment. Letters that used action language caused larger effects, particularly among Black and Hispanic individuals in Medicaid expansion states.

Leight, J., Hensly, C., Chissano, M., Safran, E., Ali, L., Dustan, D., & Jamison, J. (2022). The effects of text reminders on the use of family planning services: Evidence from a randomised controlled trial in urban Mozambique. BMJ Global Health, 7:e007862. https://doi.org/10.1136/bmjgh-2021-007862.

  • The evaluation enrolled 5,370 women in 2020 who received a clinic referral for family planning services. Evidence from this trial suggests that text message reminders are a promising nudge that increases the probability that women receive contraception.

Root, A., Connolly, C., Majors, S., Ahmed, H., & Toma, M. (2022). Electronic blood glucose monitoring impacts on provider and patient behavior. Journal of the American Medical Informatics Association, ocac069. https://doi.org/10.1093/jamia/ocac069.

  • A large randomized controlled trial tested the impact of multiple interventions to promote use of electronic blood glucose tracking on both health providers and patients. Ungated access.

Sacarny, A., Safran, E., Steffel, M., Dunham, J. R., Abili, O. D., Mohajeri, L., Oh, P. T., Sim, A., Brutcher, R. E., & Spevak, C. (2022). Effect of pharmacist email alerts on concurrent prescribing of opioids and benzodiazepines by prescribers and primary care managers: A randomized clinical trial. JAMA Health Forum, 3(9):e223378. https://doi.org/10.1001/jamahealthforum.2022.3378.

  • In this randomized clinical trial, email alerts failed to detectably reduce concurrent prescribing of opioids and benzodiazepines, which can put patients at risk of overdose. The email alerts had no statistically significant effect on patient receipt of these medications or on practitioner prescribing.

2020

Anteneh, A., Bidwell, K., Girma, W., Little, K., Wilson, N., & Workalemahu, E. (2020). Appraising praise: experimental evidence on positive framing and demand for health services. Applied Economics Letters. https://www.tandfonline.com/doi/full/10.1080/13504851.2020.1761939.

  • This study used field experiment with over 800 HIV+ female sex workers (FSWs) in Ethiopia testing the effects of providing of Praise Message phone calls on retention in antiretroviral (ART) care and adherence to ART medication. We find mixed evidence on the effects of Praise Messages, suggesting further investigation into the effects of praise or other non-informative communication on health behaviour. Ungated access.

Chaudhry, S. J., Hand, M., & Kunreuther, H. (2020). Broad bracketing for low probability events. Journal of Risk and Uncertainty, 61(3), 211–244. https://doi.org/10.1007/s11166-020-09343-4.

  • Individuals tend to underprepare for rare, catastrophic events because of biases in risk perception. A simple form of broad bracketing—presenting the cumulative probability of loss over a longer time horizon—has the potential to alleviate these barriers to risk perception and increase protective actions such as purchasing flood insurance. Across six incentive-compatible experiments with high stakes, we find that the broad bracketing effect does not disappear or change size when decisions are made from experience.

Leight, J., & Wilson, N. (2020). Framing flexible spending accounts: A large-scale field experiment on communicating the return on medical savings accounts. Health Economics, 29(2), 195–208. https://doi.org/10.1002/hec.3965.

  • This paper seeks to address this gap in the literature, reporting on a randomized controlled field experiment conducted with over 11,000 U.S federal employees in 2017 in order to evaluate the effectiveness of targeted messages designed to increase FSA contributions. Our results suggest that the provision of basic information about FSAs delivered via an emailed employee newsletter did not affect the likelihood of contribution or the contribution level. Ungated access.

Shephard, D. D., Hall, C. C., & Lamberton, C. (2020). Increasing identification of homeless students: An experimental evaluation of increased communication incorporating behavioral insights. Educational Researcher, 50(4), 239-248. https://doi.org/10.3102/0013189X20981067.

  • Across 1,732 local education agencies in New Jersey, New Mexico, and New York, we conducted a randomized controlled trial of increased email communication incorporating behavioral insights targeting homeless liaison staff in order to increase the identification of homeless students. The intervention had an impact on the mean number of identified homeless students among the treatment local education agencies. Ungated access.

2019

Chen, N., Debroy, P., Hall, S., & Le, Q. (2019, March-April). Postcards-increasing vaccination rates among elderly: U.S. Office of Evaluation Sciences and LDH Immunization Program. Louisiana Morbidity Report, 30(2):3. https://ldh.la.gov/news/.

  • This collaboration between OES and the Louisiana Department of Health Immunization Program evaluated the effect of sending a postcard reminder on elderly vaccination rates. This study showed that a postcard reminder can encourage the elderly to vaccinate, and that the timing of that reminder matters.

Leight, J., & Safran, E. (2019). Increasing immunization compliance among schools and day care centers: Evidence from a randomized controlled trial. Journal of Behavioral Public Administration, 2(2). https://doi.org/10.30636/jbpa.22.55.

  • This paper reports on the results of a randomized controlled trial in which researchers collaborated with a department of health in a mid-size city to evaluate the effectiveness of targeted communications highlighting descriptive social norms to increase immunization compliance across 700 schools.

Sacarny, A., Barnett, M. L., & Agrawal, S. (2019). New evidence on stemming low-value prescribing. NEJM Catalyst, April 10, 2019, catalyst.nejm.org/new-evidence-stemming-overprescribing/.

  • This article examines the effect that different types of messaging can have on reducing overprescribing.

Sacarny, A., Olenski, A. R., & Barnett, M. L. (2019). Association of quetiapine overuse letters with prescribing by physician peers of targeted recipients: A secondary analysis of a randomized clinical trial. JAMA Psychiatry, 76(10):1094–1095. https://doi.org/10.1001/jamapsychiatry.2019.1418.

  • A randomized clinical trial of antipsychotic overuse letters that were sent by the Centers for Medicare and Medicaid Services (CMS) to high prescribers of quetiapine reduced prescribing by targeted physicians by 16% over 2 years. This study examines whether these letters led to changes in prescribing by peers of the original physicians, which would suggest that overuse interventions can have broader effects.

2018

Sacarny, A., Barnett, M. L., & Le, J. (2018). Effect of peer comparison letters for high-volume primary care prescribers of quetiapine in older and disabled adults: a randomized clinical trial. JAMA Psychiatry, 75(10): 1003-1011. https://doi.org/10.1001/jamapsychiatry.2018.1867.

  • In this randomized clinical trial, a peer comparison letter randomized across the 5055 highest Medicare prescribers of the antipsychotic quetiapine fumarate reduced prescribing for at least 2 years. Effects were larger than those observed in existing large-scale behavioral interventions, potentially because of the content of the peer comparison letter, which mentioned the potential for a review of prescribing activity.

Yokum, D., Lauffenburger, J. C., Ghazinouri, R., & Choudhry, N. K. (2018). Letters designed with behavioural science increase influenza vaccination in Medicare beneficiaries. Nature Human Behaviour, 2(10): 743–749. https://doi.org/10.1038/s41562-018-0432-2.

  • This study used a randomized controlled trial approach to evaluate a low-cost, light-touch intervention aimed at reducing the inappropriate provision of Schedule II controlled substances in the Medicare Part D Program. Ungated access.

2017

Bowers, J., Higgins, N., Karlan, D., Tulman, S., & Zinman, J. (2017). Challenges to replication and iteration in field experiments: evidence from two direct mail shots. American Economic Review, 107(5): 462-465. https://doi.org/10.1257/aer.p20171060.

  • This study conducted field experiments to analyze the effect of two direct mailing techniques on enrollment rates in microloans to farmers. The study found that while borrowing from a government program increased the first year, the direct effect did not replicate in the second year, thus lowering the likelihood of information spillover.

Goldin, J., Homonoff, T., & Tucker-Ray, W. (2017). Retirement contribution rate nudges and plan participation: evidence from a field experiment. American Economic Review, 107(5): 456-461. https://doi.org/10.1257/aer.p20171059.

  • This field experiment analyzes how short messages can increase retirement savings by U.S. military service-members. The study finds that service-members who received a message emphasizing a low contribution rate were more likely to participate in a savings plan than were. service-members whose message emphasized a high contribution rate, or no rate at all.

Sacarny, A., Yokum, D., & Agrawal, S. (2017). Government-academic partnerships in randomized evaluations: the case of inappropriate prescribing. American Economic Review, 107(5): 466-470. https://doi.org/10.1257/aer.p20171061.

  • This study conducted several randomized letter interventions targeting high-volume prescribers of drugs that can harm patients. It utilized a continuous improvement approach that rapidly evaluates each round and uses the results to inform subsequent work.

Sacarny, A., Yokum, D., Finkelstein, A., & Agrawal, S. (2017). Medicare letters to curb overprescribing of controlled substances had no detectable effect on providers. Health Affairs, 35(3): 471-479. https://doi.org/10.1377/hlthaff.2015.1025.

  • This study used a randomized controlled trial approach to evaluate a low-cost, light-touch intervention aimed at reducing the inappropriate provision of Schedule II controlled substances in the Medicare Part D Program.

2016

Boland, A. C. (2016). Behavioral insights for better implementation in government. Public Administration Review 76, no. 4 (June 2016).

  • This article discusses the behavioral insights learned and successful interventions implemented from the Social and Behavioral Sciences Team one year after its establishment.

U.S. Department of Agriculture. Economic Research Service. (2016). USDA Microloans for Farmers: Participation Patterns and Effects of Outreach, by Sarah Tulman, Nathaniel Higgins, Robert Williams, Michael Gerling, Charles Dodson, and Bruce McWilliams. Economic Research Report No. (ERR-222). Washington DC: USDA, 2016. (accessed May 16, 2019).

  • U.S. Department of Agriculture, Farm Service Agency’s (FSA) Microloan program, launched in 2013, aims to better serve the credit needs of several types of farmers: small, beginning, veteran, and/or from historically socially disadvantaged groups (women and minorities). ERS researchers investigate the composition of Microloan borrowers and the receipt of these loans by new FSA borrowers.