Lessons from eight direct communications evaluations

What was the challenge?

Vaccination rates for many vaccines fall well below recommended levels in the United States.¹ This is especially relevant given the COVID-19 crisis, wherein a substantial fraction of the eligible population remained unvaccinated after vaccines were available.²

What did we do?

Between 2015 to 2019, we conducted eight randomized evaluations of direct communications informed by evidence-based insights from the behavioral and social sciences aimed at increasing vaccination uptake at scale. We conducted the evaluations in collaboration with a private health facility, a city department of health, a state department of health, three Veterans Health Affairs health care systems, and one operating division in the U.S. Department of Health and Human Services. The evaluations had a median sample size of 55,000 and reached diverse populations, including Veterans, the elderly, schools and day-care centers, and pregnant women.

The interventions ranged from email, postcard, letter, or social media notifications for potential vaccine recipients to a more formal report card of a school’s vaccination compliance rate for school administrators or intensive change to a hospital’s electronic health record (EHR) clinical reminders for clinicians. We’ve published the full set of interventions as “intervention packs” for each project, to share the materials and key information involved in executing these projects. The intervention packs contain materials (such as the letters, postcards, Electronic Health Record clinical reminder designs, messages etc.) to show the details of an intervention and convey the behavioral insights used in projects.

What did we learn?

An internal meta-analysis of the six evaluations that measured vaccination rates at the individual level showed an overall effect size that is positive but small, and with a confidence interval that includes the null: 0.19 percentage points (95% CI [-0.004, 0.394]). Interventions like those in our evaluations are unlikely to reliably generate effects of more than about half a percentage point. There are four key lessons learned for future vaccination efforts:

1. Behaviorally-informed direct communications can increase vaccination rates at scale but may have smaller, less reliable effects than the published literature suggests.
2. Additional evidence is needed to evaluate the cost effectiveness of behaviorally-informed direct communications relative to other interventions.
3. Rapid evaluations of vaccination uptake interventions in real-world contexts are essential for learning what works in specific contexts for populations of interest.
4. Leveraging vaccination administration systems to support randomized evaluations can make evidence building easier.

Read the summary (PDF)

Footnotes:

1. Approved by the National Vaccine Advisory Committee on September 17, 2019. (2020). 2020 National Vaccine Plan Development: Recommendations From the National Vaccine Advisory Committee. Public Health Reports, 135(2), 181-188.
   
2. Centers for Disease Control and Prevention: https://covid.cdc.gov/covid-data-tracker/#vaccinations_vacc-total-admin-rate-total