Below are selected publications resulting from OES collaborations.

All of our work is available on our website, regardless of outcome, in line with research best practices. Our first priority is to make project results accessible and actionable for agency partners. More information about publications resulting from OES collaborations can be found on our Google Scholar page.

2023

Kappes, H. B., Toma, M., Balu, R., Burnett, R., Chen, N., Johnson, R., Leight, J., Omer, S. B., Safran, E., Steffel, M., Trump, K.-S., Yokum, D., & Debroy, P. (2023). Using communication to boost vaccination: Lessons for COVID-19 from evaluations of eight large-scale programs to promote routine vaccinations. Behavioral Science & Policy, 9(1), 11-24. Available at https://doi.org/10.1177/23794607231192690.

  • The COVID-19 pandemic has added new urgency to the question of how best to motivate people to get needed vaccines. In this article, we present lessons gleaned from government evaluations of eight large randomized controlled trials of interventions that used direct communications to increase the uptake of routine vaccines. These evaluations, conducted by the U.S. General Services Administration’s Office of Evaluation Sciences (OES) before the start of the pandemic, had a median sample size of 55,000. Participating organizations deployed a variety of behaviorally informed direct communications and used administrative data to measure whether people who received the communications got vaccinated or took steps toward vaccination.

Toma, Mattie and Burnett, Russell and Debroy, Pompa and Dimant, Eugen and Liu, Jean and Safran, Elana and Saya, Uzaib and Schultz, Bill, Promoting Responsible Disposal of Opioids: A Randomized Evaluation of Behaviorally Informed Messaging Combined with a Financial Incentive (July 6, 2023). Available at http://dx.doi.org/10.2139/ssrn.4502634.

  • The opioid epidemic is a major public health problem that affects millions of people around the world. This paper addresses one challenge identified on the demand side of the epidemic: A large proportion of patients prescribed opioids lose track of their unused pills or save them for future use. Here, we report the results of a randomized controlled trial investigating an intervention aimed at encouraging patients to return unused opioid pills to the dispensing pharmacy.

2022

Debroy, P., Balu, R., Burnett, R., Johnson, R. A., Kappes, H. B., Wallace, J. M., & Marconi, V. C. (2022). “A cluster randomized controlled trial of a modified vaccination clinical reminder for primary care providers.” Health Psychology. Advance online publication.

  • We tested a provider-focused vaccination uptake intervention: a modified electronic health record clinical reminder that bundled together three adult vaccination reminders, presented patient vaccination history, and included talking points for providers to address vaccine hesitancy. Conclusion: Provider-focused interventions are a promising way to address vaccine hesitancy, but they may need to be more intensive than a modified clinical reminder to have appreciable effects on vaccination uptake.

Adam Sacarny, Elana Safran, Mary Steffel, Jacob R. Dunham, Orolo D. Abili, Lobat Mohajeri, Patricia T. Oh, Alan Sim, Robert E. Brutcher, and Christopher Spevak. “Effect of Pharmacist Email Alerts on Concurrent Prescribing of Opioids and Benzodiazepines by Prescribers and Primary Care Managers: A Randomized Clinical Trial.” JAMA Health Forum. 3(9):e223378 (2022).

  • In this randomized clinical trial, email alerts failed to detectably reduce concurrent prescribing of opioids and benzodiazepines, which can put patients at risk of overdose. The email alerts had no statistically significant effect on patient receipt of these medications or on practitioner prescribing.

Allyson Root, Christopher Connolly, Season Majors, Hassan Ahmed, and Mattie Toma. “Electronic blood glucose monitoring impacts on provider and patient behavior.” Journal of the American Medical Informatics Association, ocac069 (2022).

  • A large randomized controlled trial tested the impact of multiple interventions to promote use of electronic blood glucose tracking on both health providers and patients.

Jessica Leight, Catherine Hensly, Marcos Chissano, Elana Safran, Liza Ali, Domingos Dustan, and Julian Jamison. “The effects of text reminders on the use of family planning services: evidence from a randomised controlled trial in urban Mozambique.” BMJ Global Health 7:e007862 (2022).

  • The evaluation enrolled 5,370 women in 2020 who received a clinic referral for family planning services. Evidence from this trial suggests that text message reminders are a promising nudge that increases the probability that women receive contraception.

Andrew Feher, Daniel J. Hopkins, Elana Safran, Joshua Peck, and David Yokum. “Effectiveness of Behaviorally Informed Letters on Health Insurance Marketplace Enrollment: A Randomized Clinical Trial.” JAMA Health Forum, 3(3):e220034 (2022).

  • In this randomized clinical trial that included 744,510 individuals on the HealthCare.gov platform during the final 2 weeks of the 2015 open enrollment period, use of a single behaviorally informed letter caused a statistically significant increase in health insurance enrollment. Letters that used action language caused larger effects, particularly among Black and Hispanic individuals in Medicaid expansion states.

2021

Rekha Balu, Russell Burnett, Nuole Chen, Pompa Debroy, Rebecca A. Johnson, Heather Kappes, Jessica Leight, Saad B. Omer, Elana Safran, Mary Steffel, Mattie Toma, Kris-Stella Trump, and David Yokum. “Lessons for Covid-19 Vaccination from Eight Federal Government Direct Communication Evaluations.” Behavioral Science & Policy, Forthcoming, Available at SSRN: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3967610.

  • We discuss eight randomized evaluations intended to increase vaccination uptake conducted by the US General Services Administration’s Office of Evaluation Sciences (OES). Two studies yielded statistically significant increases, of 0.59 and 0.16 percentage points. The other six were not statistically significant, although the studies were powered to detect effect sizes in line with published research. This work highlights the likely effects of government communications and demonstrates the value of conducting rapid evaluations to support COVID-19 vaccination efforts.

2020

Jeffrey Hemmeter, John Phillips, Elana Safran, and Nicholas Wilson. “Communicating Program Eligibility: A Supplemental Security Income (SSI) Field Experiment.” (2020).

  • We conducted a randomized controlled direct mail field experiment with 4,016,461 individuals to test several key hypotheses about why take-up of this program is low. We find that communicating likely eligibility in a basic letter generated substantial increases in take-up in relative terms, yet the application rate in our study sample during the full nine-month follow-up period remained at less than 7%. Adding behaviorally-informed statements increased the effectiveness of these communications.

Daniel D. Shephard, Crystal C. Hall, and Cait Lamberton. “Increasing Identification of Homeless Students: An Experimental Evaluation of Increased Communication Incorporating Behavioral Insights.” Educational Researcher, 50 (4), 239-248 (2020).

  • Across 1,732 local education agencies in New Jersey, New Mexico, and New York, we conducted a randomized controlled trial of increased email communication incorporating behavioral insights targeting homeless liaison staff in order to increase the identification of homeless students. The intervention had an impact on the mean number of identified homeless students among the treatment local education agencies.

Aderaw Anteneh, Kelly Bidwell, Woldemariam Girma, Kristen Little, Nicholas Wilson, and Endale Workalemahu. “Appraising praise: experimental evidence on positive framing and demand for health services.” Applied Economics Letters, (2020).

  • This study used field experiment with over 800 HIV+ female sex workers (FSWs) in Ethiopia testing the effects of providing of Praise Message phone calls on retention in antiretroviral (ART) care and adherence to ART medication. We find mixed evidence on the effects of Praise Messages, suggesting further investigation into the effects of praise or other non-informative communication on health behaviour.

Shereen J. Chaudhry, Michael Hand, and Howard Kunreuther. “Broad bracketing for low probability events.” Journal of Risk and Uncertainty, 61 (3), 211–244 (2020).

  • Individuals tend to underprepare for rare, catastrophic events because of biases in risk perception. A simple form of broad bracketing—presenting the cumulative probability of loss over a longer time horizon—has the potential to alleviate these barriers to risk perception and increase protective actions such as purchasing flood insurance. Across six incentive-compatible experiments with high stakes, we find that the broad bracketing effect does not disappear or change size when decisions are made from experience.

2019

Adam Sacarny, Andrew R. Olenski, and Michael L. Barnett. Association of Quetiapine Overuse Letters With Prescribing by Physician Peers of Targeted Recipients: A Secondary Analysis of a Randomized Clinical Trial. JAMA Psychiatry, June 4, 2019.

  • A randomized clinical trial of antipsychotic overuse letters that were sent by the Centers for Medicare and Medicaid Services (CMS) to high prescribers of quetiapine reduced prescribing by targeted physicians by 16% over 2 years. This study examines whether these letters led to changes in prescribing by peers of the original physicians, which would suggest that overuse interventions can have broader effects.

Adam Sacarny, Michael L. Barnett, and Shantanu Agrawal. New Evidence on Stemming Low-Value Prescribing. NEJM Catalyst, April 10, 2019, catalyst.nejm.org/new-evidence-stemming-overprescribing/.

  • This article examines the effect that different types of messaging can have on reducing overprescribing.

Jessica Leight and Nicholas Wilson. Framing Flexible Spending Accounts: A Large-Scale Field Experiment on Communicating the Return on Medical Savings Accounts. Health Economics 29, no. 2 (2019): 195–208.

  • This paper seeks to address this gap in the literature, reporting on a randomized controlled field experiment conducted with over 11,000 U.S federal employees in 2017 in order to evaluate the effectiveness of targeted messages designed to increase FSA contributions. Our results suggest that the provision of basic information about FSAs delivered via an emailed employee newsletter did not affect the likelihood of contribution or the contribution level.

Jessica Leight and Elana Safran. Increasing immunization compliance among schools and day care centers: Evidence from a randomized controlled trial. Journal of Behavioral Public Administration 2, no. 2 (2019).

  • This paper reports on the results of a randomized controlled trial in which researchers collaborated with a department of health in a mid-size city to evaluate the effectiveness of targeted communications highlighting descriptive social norms to increase immunization compliance across 700 schools.

Noule Chen, Pompa Debroy, Stacy Hall, and Quan Le. Postcards-increasing vaccination rates among elderly: U.S. Office of Evaluation Sciences and LDH Immunization Program. Louisiana Morbidity Report 30 no. 2 (April 2019): 3-7.

  • This collaboration between OES and the Louisiana Department of Health Immunization Program evaluated the effect of sending a postcard reminder on elderly vaccination rates. This study showed that a postcard reminder can encourage the elderly to vaccinate, and that the timing of that reminder matters.

2018

Adam Sacarny, Michael L. Barnett, and Jackson Le. Effect of peer comparison letters for high-volume primary care prescribers of quetiapine in older and disabled adults: a randomized clinical trial. JAMA Psychiatry 75 no. 10 (October 2018): 1003-1011.

  • In this randomized clinical trial, a peer comparison letter randomized across the 5055 highest Medicare prescribers of the antipsychotic quetiapine fumarate reduced prescribing for at least 2 years. Effects were larger than those observed in existing large-scale behavioral interventions, potentially because of the content of the peer comparison letter, which mentioned the potential for a review of prescribing activity.

David Yokum, Julie C. Lauffenburger, Roya Ghazinouri, and Niteesh K. Choudhry. Letters designed with behavioural science increase influenza vaccination in Medicare beneficiaries. Nature Human Behaviour 2 (2018): 743–749.

  • This study used a randomized controlled trial approach to evaluate a low-cost, light-touch intervention aimed at reducing the inappropriate provision of Schedule II controlled substances in the Medicare Part D Program.

2017

Adam Sacarny, David Yokum, Amy Finkelstein, and Shantanu Agrawal. Medicare letters to curb overprescribing of controlled substances had no detectable effect on providers. Health Affairs 35 no. 3 (2017): 471-479.

  • This study used a randomized controlled trial approach to evaluate a low-cost, light-touch intervention aimed at reducing the inappropriate provision of Schedule II controlled substances in the Medicare Part D Program.

Adam Sacarny, David Yokum, and Shantanu Agrawal. Government-academic partnerships in randomized evaluations: the case of inappropriate prescribing, American Economic Review 107, no. 5, (2017): 466-470.

  • This study conducted several randomized letter interventions targeting high-volume prescribers of drugs that can harm patients. It utilized a continuous improvement approach that rapidly evaluates each round and uses the results to inform subsequent work.

Jacob Goldin, Tatiana Homonoff, and William Tucker-Ray. Retirement contribution rate nudges and plan participation: evidence from a field experiment, American Economic Review 107, no. 5 (2017): 456-461.

  • This field experiment analyzes how short messages can increase retirement savings by U.S. military service-members. The study finds that service-members who received a message emphasizing a low contribution rate were more likely to participate in a savings plan than were. service-members whose message emphasized a high contribution rate, or no rate at all.

Jake Bowers, Nathaniel Higgins, Dean Karlan, Sarah Tulman, and Jonathan Zinman. Challenges to replication and iteration in field experiments: evidence from two direct mail shots, American Economic Review 107, no. 5: (2017): 462-465.

  • This study conducted field experiments to analyze the effect of two direct mailing techniques on enrollment rates in microloans to farmers. The study found that while borrowing from a government program increased the first year, the direct effect did not replicate in the second year, thus lowering the likelihood of information spillover.

2016

Amira Choueiki Boland. Behavioral insights for better implementation in government. Public Administration Review 76, no. 4 (June 2016).

  • This article discusses the behavioral insights learned and successful interventions implemented from the Social and Behavioral Sciences Team one year after its establishment.

U.S. Department of Agriculture. Economic Research Service. USDA Microloans for Farmers: Participation Patterns and Effects of Outreach, by Sarah Tulman, Nathaniel Higgins, Robert Williams, Michael Gerling, Charles Dodson, and Bruce McWilliams. Economic Research Report No. (ERR-222). Washington DC: USDA, 2016. (accessed May 16, 2019).

  • U.S. Department of Agriculture, Farm Service Agency’s (FSA) Microloan program, launched in 2013, aims to better serve the credit needs of several types of farmers: small, beginning, veteran, and/or from historically socially disadvantaged groups (women and minorities). ERS researchers investigate the composition of Microloan borrowers and the receipt of these loans by new FSA borrowers.