1. What are the benefits of a learning agenda?
2. What types of questions go into a learning agenda?
3. Who should be involved in the learning agenda development process?
4. How do I get leadership input and support?
5. What is the process for developing a learning agenda?
6. How does stakeholder engagement inform question development?
7. How do I write my agency's learning agenda?
8. Where can I find additional resources on learning agendas?
A learning agenda can be a useful coordination tool for identifying knowledge gaps and answer a range of priority questions within budget and policy timeframes.1 How can a learning agenda support your agency in achieving its mission?
OMB M-19-23 Reference
What is a learning agenda (evidence-building plan)?
[A learning agenda is] is a systematic plan for identifying and addressing priority questions relevant to the programs, policies, and regulations of the agency… to support effective and efficient agency functioning.
Benefits of a Learning Agenda
Learning agendas can have many benefits for agencies. Specifically, the plans can be used to:
Foster a culture of learning and continuous improvement. Learning agenda development and implementation includes frequent internal and external stakeholder engagement. As described throughout this toolkit, the Evaluation Officer (EO) and related staff can use engagement opportunities to educate and encourage stakeholders to build and use evidence.
The EO could establish an Agency-Specific Evidence Community of Practice to foster ideas, share practices, and build a culture of learning.
Engage stakeholders in evidence planning and building to help achieve an agency’s mission. The learning agenda process provides opportunities to work with both internal and external stakeholders to identify priority areas. By engaging stakeholders early and often, an agency can continuously build evidence that can be used to improve strategy and operations.
The EO and related staff could facilitate a series of listening sessions to hear potential priority questions.
Create a dynamic and adaptive Strategic Plan. Priority questions in the learning agenda may align to an agency’s goals or objectives. Alignment to the Strategic Plan can help agencies build evidence on leading strategies to accomplish goals and objectives.
The EO and Performance Improvement Officer (PIO) should collaborate during the annual strategic review to discuss priority questions with progress towards strategic goals and objectives.
Help develop other major agency plans. As explained in Section 5: Developing a Learning Agenda, the learning agenda can support a variety of major plans and activities, including the Unified Agenda of Federal Regulatory and Deregulatory Actions and requirements from the Government Performance and Results Act (GPRA) Modernization Act and Program Management Improvement and Accountability Act.
The EO and Program Management Improvement Officer (PMIO) could collaborate to use the annual portfolio reviews, if applicable, as a tool to help answer priority questions.
Establish a framework for weaving evidence into the budget cycle. Learning agenda questions can support programs, policies, regulations, and other needs. A well-designed schedule of learning activities can provide timely information for budgeting decisions and lead to increases in resource efficiency.
The EO could collaborate with the Chief Financial Officer (CFO) to modify budget instructions to include evidence requirements and prioritize evidence aligned to learning agenda questions.
Coordinate research, analysis, and evaluation among sub-agencies. The EO and related staff can use the learning agenda process to bridge sub-agency silos. This includes eliminating duplicate efforts; sharing methods, resources, and leading practices; and identifying opportunities for crosscutting learning activities.
The EO and related staff could involve diverse sub-agency representation in a working group or core team to decide which questions are included in the agency-wide learning agenda.
Plan congressionally mandated evaluations and address recommendations from federal watchdog agency reports. As appropriate, the learning agenda can provide a framework for developing a plan to conduct required evaluations, close out recommendations, remediate issues, and offer supporting evidence.
The agency can review watchdog report recommendations and work with the EO to determine areas where evaluation could address the recommendation.
Publicly identify learning priorities. The learning agenda will be a public document. This promotes transparency and may allow external stakeholders an opportunity to build evidence in areas of interest to an agency, sometimes at little or no cost.
A learning agenda priority question could be shared on Challenge.gov, a crowdsourcing web platform where members of the public compete to offer ideas and solutions.
Initial Steps to Developing a Learning Agenda
The EO and related staff can use the resources in this toolkit to design and execute a learning agenda process best suited to agency needs. Here are a few examples of steps your agency can take now to advance the learning agenda process:
Review your agency’s major plans to understand its evidence base.
Your agency may already use evidence to inform strategy and operations. For example, several agency Strategic Plans include sections on program evaluation or evidence-based policymaking and decision-making. The learning agenda process can build off these existing activities.
Meet with key decision makers in your agency.
Senior leaders are critical and required stakeholders in the learning agenda process. In an initial meeting with senior leadership, the EO and related staff could discuss the benefits listed above and how the learning agenda will be used. Consider establishing a formal Evidence Leadership Council with senior leaders, including the agency Chief Data Officer (CDO) and Statistical Official (SO) (the other designated officials established by the Evidence Act). The council or working group could receive updates on the process and deliberate on the plan’s components. Download the attachment at the top of this page for a one-page Learning Agenda Overview to share with agency staff.
Establish a cross-functional leadership team.
The EO should consider establishing formal, recurring meetings with the CDO and SO. As described in Section 5: Developing a Learning Agenda, other officials may be valuable to include in the recurring meetings, such as the Chief Risk Officer, PIO, or PMIO. Where possible and appropriate, these officials should join meetings with senior leadership together. When discussing agency issues, it is valuable to have the diverse perspectives of strategy, evidence, data, and statistical leaders.
Learn about leading practices in learning agenda development and use.
See Section 8: Additional Resources for links to previous learning agendas and similar Commented [HA3]: Include hyperlink to section Commented [HA4]: Include hyperlink to section evidence-building plans developed by federal agencies, links to thought leadership, and a variety of trainings.
Continue reading the toolkit for more tips, templates, suggested processes, and examples to help you complete your agency’s learning agenda.
A learning agenda includes “priority questions” for the next four years. What is a “priority question” and what are some examples?
OMB M-19-23 Reference
An agency learning agenda addresses priority questions (i.e., questions relevant for programmatic, operational, regulatory, or policy decision-making) across the entire agency… Agencies should identify priority questions that, when answered, will have the biggest impact on agency functioning and performance..
Overview of Priority Questions
An agency learning agenda includes priority questions relevant to programs, policies, and regulations. What is a priority question? These are questions that, when answered, have the biggest impact on your agency’s functioning and performance. How many questions meet the criteria of “biggest impact” is up to your agency, and the number of questions may be influenced by agency size or capacity.
Stakeholder engagement is required to develop the learning agenda per the Evidence Act. Stakeholders can offer valuable input on potential priority questions that will have the biggest impact on agency functioning and performance. This toolkit offers strategies for identifying stakeholders (Section 3: Engaging Stakeholders), consulting leadership (Section 4: Encouraging Leadership Buy-In), timing stakeholder engagement activities (Section 5: Developing a Learning Agenda), and synthesizing stakeholder input into a set of priority questions to include in your agency’s learning agenda (Section 6: Using Stakeholder Input). Throughout the stakeholder engagement process, consider the following tips about priority questions:
Where appropriate, start with an overarching question and develop specific, answerable questions. Useful learning agenda questions do not need to be narrow in scope. However, a question may be too broad to be answerable. Below is an example of an overarching question from a fictitious Department of Societal Betterment, whose agency mission is to reduce societal nuisances. The question is followed by series of sub-questions that are specific and answerable:
- Overarching question: “How can we reduce public nuisances?” This question could be included in a learning agenda, but it does not specify which of the many types of nuisances will be the focus for learning.
- Specific and answerable questions: The level of specificity can vary for an answerable question. For example, the following three questions may all be considered answerable for the Department of Societal Betterment. The first looks OMB M-19-23 Reference An agency learning agenda addresses priority questions (i.e., questions relevant for programmatic, operational, regulatory, or policy decision-making) across the entire agency… Agencies should identify priority questions that, when answered, will have the biggest impact on agency functioning and performance. Commented [HA5]: Include hyperlink to section Commented [HA6]: Include hyperlink to section Commented [HA7]: Include hyperlink to section Commented [HA8]: Include hyperlink to section across agency programs, the second focuses on the efficacy of one new program, and the last looks within the new program. Most importantly, it is possible to write a description of evidence-building activities that will answer each question.
- Has the policy setting a $300 fine for littering led to an increase or decrease in littering in the last 10 years?
- Does new signage emphasizing pride in our public places reduce the incidence of littering?
- Which message on the new signage, among the three versions, is most effective in encouraging people not to litter?
Learning agenda questions do not need to be answerable with one study. Important questions often require sub-questions and multiple studies or analyses, typically building on each other, to create a useful and actionable body of evidence.
Consider how each question could be measured. Can the question be answered with qualitative or quantitative data? If yes, does the data exist, or does your agency need to plan new data collection efforts? Without supporting evidence, the answer to the question is unlikely to inform agency decisions.
Include questions that draw on different components of evidence. As your agency formulates questions for its learning agenda, think ahead about evidence-building activities. How could policy analysis, program evaluation, foundational fact-finding, or performance measurement help answer your priority questions? As shown to the right, these are the four components of evidence described in OMB M-19-23. See the box below for a subset of the Small Business Administration’s (SBA) learning agenda questions, organized by evidence type.
OMB M-19-23 Reference
How the Small Business Administration’s Enterprise Learning Agenda uses all Four Components of Evidence
The SBA develops an Enterprise Learning Agenda (ELA) to empower program managers to achieve their priorities while fostering an environment of continuous learning. The ELA is a five-year plan that aligns with the Agency’s goals outlined in the FY 2018-2022 Strategic Plan. The ELA includes questions that cover the different components of evidence to allow for evidence building across a spectrum of methods. The following questions are aligned to the strategic goal, “Support small business revenue and job growth.”
- Foundational fact-finding: What types of jobs do SBA loans create?
- Policy analysis: What regulatory or policy improvements could be made to help SBA programs create more jobs?
- Performance measurement: Are program grantees meeting their milestones?
- Program evaluation: How does technical assistance impact loan performance?
Defining “Short-Term” and “Long-Term” Questions
Agencies should include short-term and long-term questions in their learning agendas, including definitions of the timeframes (OMB M-19-23). Consider the following factors when defining what “short-term” and “long-term” mean for your agency:
- Agency capacity: Consider referencing your agency’s Capacity Assessment, if completed, to understand in-house capabilities. If a question will require independent, contracted evaluations, consider the length of the procurement process in addition to time to complete to the studies.
- Decision-making timelines: Evidence-building activities such as evaluations may require time for funding or staff resources to be allocated, even if conducted in-house. Consider when funding is set aside for evaluations, as well as the period during which the evidence will be used. For example, if your agency updates a specific policy every five years, the definition of “long-term” might align to that timeline.
- Order of operations: Before answering a long-term question, a series of shortterm questions may need to be answered first. When defining “long-term,” consider the timeline for answering consecutive short-term questions.
For more information and examples of short-term and long-term questions, please refer to Table B.1: Example of Potential Learning Agenda Questions on page 17 of OMB M-19-23.
Sample Questions by Mission Area and Agency Function
Some agencies are primarily grant-making organizations, others are operational, and many function in multiple mission areas. To assist with priority question brainstorming, see the table below for hypothetical learning agenda questions by mission area and function
|Mission Area or Agency Function||Example Questions|
|Law Enforcement, Defense, and Security||
|Research and Discovery||
Frequent stakeholder engagement is critical to the success of a learning agenda. Who are the most important stakeholders to your agency? How will you engage them?
OMB M-19-23 Reference
The purpose of stakeholder engagement is to make sure that the learning agenda addresses questions that are relevant, salient, and meaningful to those with direct interests in the agency’s functions, and that the learning that results resonates with stakeholders.
One way to frame your thinking around stakeholders is to categorize them broadly into three types: internal, external (governmental), and external (nongovernmental).2 This framework can be useful in determining the different forms of outreach and engagement needed to gather timely and relevant feedback. All three groups can offer important insights into the current state of evidence, knowledge gaps, and unmet needs. In the table below, stakeholders in green are required per the Foundations for EvidenceBased Policymaking Act of 2018 (Evidence Act, Sec. 312.C 5 USC 312. Agency evidence-building plan). The stakeholders in black should be consulted per OMB M-19-23.
|Internal||External (Governmental)||External (Nongovernmental)|
Privacy and Information Law and Policy Personnel
Evaluation, Statistics, Analysis, Data, Enterprise Risk Management, and Performance Unit Personnel
State and Local Governments
Relevant Federal Agencies
Office of Management and Budget (OMB)
Researchers and Academics
Federal Grant Recipients
Industry and Trade Groups
Relevant Non-Governmental Organizations and Non-Profits Program Beneficiaries
When identifying internal stakeholders, it may be helpful to focus on utilization – in other words, who will utilize the evidence that comes out of the learning agenda? Utilization can exist in multiple dimensions, such as programs (who could prioritize certain interventions over others based on evidence?) policies, or regulations (who could act on a body of evidence to make policy or regulatory decisions?). Think about the staff in your agency who have the opportunity to make evidence-based decisions.
Internal stakeholders may include senior leadership and Evidence Act officials (Chief Data Officer and Statistical Official) or other officials (Performance Improvement Officer, Chief Risk Officer, Program Management Improvement Officer, or leaders in regulations or privacy and information law). Consider meeting with these individuals early in the process to understand their desired roles and expectations. For example, senior leadership may request to make the final decisions on which priority questions are included in the published learning agenda.
Also consider the subject matter expertise that exists within your agency. Keep in mind that agency staff will have varying degrees of capacity and time. When creating meeting agendas in Worksheet #2 of the Learning Agenda Workbook (available for download in the tab to the right), consider dedicating time to discuss expectations for their involvement.
External Stakeholders (Governmental)
When identifying external government stakeholders, consider asking two questions. First, where are there overlaps in your agency’s mission at the local, state, and Federal levels? Coordination with agencies performing similar functions may increase the evidence base and could lead to opportunities for evidence-building partnerships. Second, where are there existing supports and expertise in evidence building and use across the federal government? You could engage experienced EOs for mentorship, request support or advice from OMB or the General Services Administration (GSA), or invite leaders in the field to speak at an agency event.
External Stakeholders (Nongovernmental)
Your internal stakeholder network described above may have existing relationships and communication channels with relevant nongovernmental stakeholders (e.g., academic institutions, the public, non-profits, consumer groups, advocacy groups, or grantee organizations). By engaging with a variety of external, nongovernmental stakeholders, your agency can signal that it values broad feedback and participation in its evidencebuilding activities.
When identifying external nongovernmental stakeholders, consider those who can offer diverse perspectives, knowledge, and experiences on issues relevant to your agency’s mission and operations. External stakeholders can help identify knowledge gaps, assumptions, and risks while also raising potential priority questions relevant to your agency’s operating environment. Through external stakeholder engagement, your agency can document unanswered research questions in the field and discover new data or analytical tools to help answer priority questions.
Learning Agenda Core Team
If the learning agenda core team cannot be adequately staffed with in-house resources, you can use other resources such as contractor support, Intergovernmental Personnel Act (IPA) agreements, or detailees from experienced Federal agencies. In Worksheet 1 of the Learning Agenda Workbook document the core team and their roles and responsibilities in Table 1.
Consider inviting an agency leadership champion to join core team meetings. Leadership input is required in developing the learning agenda (see Section 4: Encouraging Leadership Buy-In). A champion can offer ongoing feedback and may be able to make decisions outside of the team’s authority.
One alternative to forming a new team is to expand the scope of an existing group to include learning agenda development. For example, many agencies develop crossfunctional strategic planning working groups with diverse representation from subagencies. A subset of this group could be used to form a core team, or the group itself could develop or contribute to the learning agenda.
How the United States Agency for International Development Forms a Learning Agenda Technical Team
At USAID, a well-rounded team is comprised of the following skills. Due to resource constraints, individuals may span multiple roles.
- Project management: The learning agenda process requires a leader to plan, design, execute, and monitor the necessary meetings and follow-up activities. A project manager can engage with agency leadership for support and guidance, and monitor implementation to ensure objectives are being met. This individual should also ensure that the process is aligned to the agency’s strategic planning, performance management, and budgeting cycles.
- Learning-focused meeting facilitation: The learning agenda process will include many meetings with internal and external stakeholders, including senior leadership. A facilitator will stimulate conversation, leverage learning opportunities, and encourage participation from diverse stakeholders.
- Methodological and analytical skills: Learning agenda questions must be answerable. Individuals with methodological and analytical skills will provide valuable input as to the feasibility and cost of answering proposed questions.
- Subject matter expertise: The team should include individuals who can assist with identifying evidence and learning needs, and who understand how the answers to learning agenda questions will be used. Agencies are strongly encouraged to involve program staff who will use evidence in daily operations.
- Organizational learning and knowledge management skills: To promote utilization, identify team members who understand and represent the needs, expectations and constraints of end users. These individuals can identify opportunities or entry points to integrate learning products in the agency’s work. Individuals with organizational learning and knowledge management skills will advise on how learning products can be user friendly, timely, and shared with the right audience.
- Communication skills: Throughout the stakeholder engagement process, it is important to utilize a communications professional who crafts clear and consistent messaging that supports internalization of evidence generated from the learning agenda. This individual may also play a role in designing the action plan to disseminate and use evidence generated from the learning products.
If you form a learning agenda core team, meet with the team to complete Activity 1 in the Learning Agenda Workbook. The activity will help your team fill out the stakeholder matrix in Table 2. If you do not have a core team, consider using the guidance below to complete Table 2 to the best of your knowledge.
An Overview of Stakeholder Engagement Activities
There is no universal approach to stakeholder engagement – the methods will vary by stakeholder group and the agency’s needs, resources, and structure. The type of activity should also take into consideration the information you are trying to gather from the stakeholder group (e.g., recommendations for priority questions, or sharing questions for feedback to ensure that they resonate with stakeholders). Consider some examples below, keeping in mind that the Paperwork Reduction Act could apply to your information collection method:
- Email campaign
- Individual interviews
- Requests for public comment
In Section 5: Developing a Learning Agenda, the EO and related staff will design the learning agenda process. This includes when and how to engage stakeholders to get the most out of their inputs. To assist with planning stakeholder engagement meeting content, Worksheet 2 of the Learning Agenda Workbook includes sample meeting outcomes, discussion topics, and tips for consideration. Reference Worksheet 2 throughout Section 3: Engaging Stakeholders, Section 4: Encouraging Leadership Buy-In, and Section 5: Developing a Learning Agenda.
As noted above, senior leadership and program managers are very important stakeholders in the learning agenda process. The following sample meeting agendas include questions you could ask senior leadership or program managers during an initial meeting about your agency’s learning agenda. The EO and related staff could also adapt the agendas below in a group setting by selecting a sub-set of the questions listed.
Audience: Senior Leadership
|Agenda Item||Suggested Time||Questions to Consider|
|Evidence Act Overview and Requirements||5 minutes||
|Learning Agenda Overview||5 minutes||
|Learning Agenda Process and Timeline||10 minutes||
|Preliminary Thoughts on Learning Agenda Content||10 minutes||
Audience: Program Manager
|Agenda Item||Suggested Time||Questions to Consider|
|Learning Agenda Overview||5 minutes||
|Understanding the Program’s Evidence Base||20 minutes||
|Planned or Interest in EvidenceBuilding Activities Learning more about your Program||20 minutes||
|Stakeholder Engagement||15 minutes||
Leadership buy-in and input throughout the learning agenda process strengthens an agency’s culture of learning and continuous improvement. How can you encourage leadership to be champions of your agency’s learning agenda?
OMB M-19-23 Reference
Agency leadership should be engaged in the development and implementation of the learning agenda, as their buy-in and support for determining what highpriority questions should be included, which stakeholders should inform the process, and which staff and data assets will be needed to fulfill the learning agenda, are critical to success.
More broadly than the learning agenda, the Evaluation Officer (EO) and related staff can signal to leadership that a culture of learning and continuous improvement starts at the top. To transform an agency’s culture, the EO and related staff need to be supported by champions who exemplify a commitment to building and using evidence in their work. A learning culture is central to a meaningful and effective learning agenda.
General Tips for Gaining Leadership Input and Support
When designing your agency’s learning agenda process, consider the following tips for how and when to engage leadership.
Gather input from leadership upfront. You can lay the foundation for leadership buyin by asking how the learning agenda could be useful to them. Leadership engagement is a requirement per the Foundations for Evidence-Based Policymaking Act of 2018 (Evidence Act), but with many initiatives competing for leadership attention, it may be challenging to find an opportunity for engagement. Nonetheless, look for ways to engage the most senior leadership possible, up to and including the agency head. A sample meeting agenda is included at the bottom of this section to help you facilitate conversations with leadership, and additional tips can be found on Worksheet 2 of the Learning Agenda Workbook (available for download in the tab to the right). Also, see Section 1: Purpose and Benefits for an overview of the benefits of a learning agenda.
Engage leadership at the right level of detail. Agency leaders do not need to become experts on the Evidence Act or learning agendas to provide input on priority questions, key stakeholders, or learning activities. Think about how to communicate the right level of detail to leadership, emphasizing the big picture.
Make connections between learning agendas and leadership priorities. It is important to frame the learning agenda process in ways that will resonate with leadership. Consider starting with a review of your agency’s Strategic Plan and leadership memos. When requesting a meeting and setting the agenda, present the learning agenda as a coordination tool to help support leadership goals and priorities. For example, if your leadership strongly values innovation, emphasize that the learning agenda can support the agency in building evidence on innovative solutions to challenges. Demonstrate how the learning agenda will help clarify where problems exist, encourage new approaches to problem-solving, and prompt the development of credible evidence around the efficacy of those approaches.
Share success stories from other agencies. Help leadership understand why a learning agenda is useful by illustrating how other organizations have used learning agendas to improve performance or increase return on their investments. For example, the Small Business Administration organized its learning agenda around its strategic goals, an approach that helped the agency determine how well its technical assistance program for small businesses was helping those businesses grow. You can find more information about how agencies have implemented prior learning agendas in Section 8: Additional Resources.
Regularly report progress to leadership. A frequent cadence of progress reports to leadership maintains their buy-in and support. Topics of discussion might include the status of learning agenda development or results from evidence-building activities that shed light on priority issues and challenges. Consider incorporating these updates into existing, regular meetings with leadership.
Designing a Meeting with Leadership
When designing a meeting agenda using Worksheet 2 of the Learning Agenda Workbook, consider the tips described above. For your first meeting, you might focus on getting initial input into your agency’s priority learning agenda questions. Following internal and external stakeholder engagement, a future meeting with leadership could include a set of draft priority questions and a request for feedback. Each meeting with leadership should have a clear set of objectives and desired outcomes. The sample meeting agenda below is designed to elicit priority questions or other learning agenda content from leadership.
|Agenda Item||Suggested Time||Questions to Consider|
|Evidence Act Overview and an Agency Success Story||5 minutes||
|Learning Agenda Overview||5 minutes||
|Learning Agenda Process and Timeline||10 minutes||
|Preliminary Thoughts on Learning Agenda Content||10 minutes||
Agencies have the flexibility to design their learning agendas in a format that addresses their unique needs. How will your approach ensure that the learning agenda is a living document for continuous learning and improvement?
This section offers suggestions for the Evaluation Officer (EO) and related staff on how to develop a tailored learning agenda process to meet your agency’s needs. The primary resource for this section is the Master Checklist, which can be downloaded on the tab to the right. The Master Checklist provides high-level required activities and some suggested activities to develop your agency’s learning agenda. For more detailed project planning, please reference Worksheet #3 of the Learning Agenda Workbook (available for download in the tab to the right).
Listen for questions at every opportunity. In addition to a formal learning agenda process, encourage the learning agenda core team or key stakeholders to model the culture of learning and continuous improvement you are striving to build (see Section 3: Engaging Stakeholders). When issues are discussed in any agency meetings, consider asking yourself, “Could that be a learning agenda question?”
Incorporate iterative and ongoing stakeholder engagement. Regardless of the size and scope of outreach and engagement efforts, ongoing stakeholder engagement is critical to fostering a culture of learning and continuous improvement.
Planning Stakeholder Engagement Activities
In Section 3: Engaging Stakeholders, the EO and related staff identified key stakeholders for the learning agenda process using Worksheet #1 of the Learning Agenda Workbook. Consider how you will be strategic and deliberate to engage stakeholders while efficiently using your available resources. This includes the timing and length of engagement activities, as well as the number of stakeholders you will engage with at a time. In initial meetings, consider establishing recurring communications. To maintain a learning agenda as a living document, stakeholder engagement must be iterative and ongoing.
Stakeholder engagement is also an opportunity to educate others about evidence. Level of expertise on evidence should not preclude anyone from voicing their opinions on the agency’s priority questions and learning activities. Stakeholder engagement can cultivate curiosity and encourage participation in the agency’s learning culture. You can signal to agency staff that the learning agenda and other evidence-building activities are not simply compliance exercises, but tools that will ultimately help them to improve the effectiveness of their programs.
For guidance on individual stakeholder meetings, see Worksheet #2 of the Learning Agenda Workbook to design meeting agendas. Additional meeting styles, facilitation techniques, and tips are available in the USAID resource, Designing and Facilitating Learning-Focused Meetings. To plan the schedule and cadence of all stakeholder engagement activities, refer to Worksheet #3 of the Learning Agenda Workbook.
OMB M-19-23 Reference
Because this interim learning agenda is being developed during the last year of an agency’s Strategic Plan, the interim learning agenda should be written to address short-term priority questions that can be answered more immediately and long-term questions that are directly linked to the agency’s core statutory mission and therefore are unlikely to differ substantively from one strategic planning cycle to the next.
Alignment with Existing Plans and Processes
The learning agenda is a section of the Strategic Plan. While agencies are encouraged to align the learning agenda process with the strategic planning process, it is not required. Alignment with the Strategic Plan, or other major agency plans, can help streamline requirements, optimize available resources, and improve your understanding of agency priorities. The laws, plans, and activities described below are a few examples the EO should consider reviewing for alignment with the learning agenda process.
The first table of plans and processes fall under the GPRA Modernization Act (GPRAMA) of 2010. GPRAMA requires quarterly performance assessments of government programs, provides updated guidance on agency Strategic Plans, and establishes the role of Performance Improvement Officer (PIO) and the Performance Improvement Council. The PIO is responsible for Office of Management and Budget (OMB) submissions and final publication of all plans produced under GPRAMA. The table below includes an overview of each plan and a list of questions the EO and PIO may want discuss for how each plan or process could align with the learning agenda.
In addition to the list below, your agency may align its learning agenda process to fiveyear and/or annual research plans, or cycles in policy analysis, regulatory planning, or budget development. It is important to consider your agency’s processes across all four components of evidence. Especially for agencies with limited resources, an EO can take advantage of this wealth of information for potential learning agenda content.
|What is it?||A Strategic Plan establishes the goals, objectives, and strategies for an agency over a five-year period. Beginning in FY 2021, Strategic Plans will include a section with the agency’s learning agenda and Capacity Assessment|
|Questions for the EO and PIO||
|What is it?||Using the agency Strategic Plan, agency leaders assess progress on mission, service, stewardship, and cross-cutting strategic objectives. This process includes identification of areas where additional evaluation, other studies, or analyses of performance data are needed to determine effectiveness or set priorities.|
|Questions for the EO and PIO||
|Annual Performance Plan (APP)|
|What is it?||The APP is a description of the strategies necessary to achieve a desired level of performance in the next fiscal year, including the key performance targets and milestones that will be accomplished.|
|Questions for the EO and PIO||
|Quarterly Performance Reviews|
|What is it?||Performance management reviews are routine (at least quarterly) and datadriven reviews led by agency leaders on a limited set of the agency’s performance improvement priorities. The frequent reviews provide a mechanism for agency leaders to keep an agency focused on an identified set of priorities, diagnose problems, learn from experience, and determine next steps to increase performance and productivity.|
|Questions for the EO and PIO||
|Project Management Improvement Accountability Act (PMIAA) of 2016|
|What is it?||The PMIAA aims to improve program and project management practices within the federal government by requiring government-wide standards and policies for program management and establishing a new interagency council.|
|Who do I talk to?||Program Management Improvement Officer (PMIO)|
|Questions to Consider||
|Geospatial Data Act (GDA) of 2018|
|What is it?||The GDA promotes stronger partnerships between the federal government, state and local governments, and the private sector to advance the National Spatial Data Infrastructure. It includes policy and guidance to empower the use of geospatial data and technology to spur economic growth, advance scientific research, and improve public health, environmental protection, and other services.|
|Who do I talk to?||Chief Data Officer|
|Questions to Consider||How can the learning agenda process help launch or improve publicprivate partnerships for the use of quality scientific and other data in policymaking? Example: Priority questions requiring access to geospatial data can encourage leadership to invest in a data-sharing partnership.|
|Federal Managers Financial Integrity Act (FMFIA) of 1982 / Updated OMB Circular A-123 (2016)|
|What is it?||The FMFIA requires federal agencies to establish internal accounting and administrative controls. OMB Circular A-123 includes requirements and guidance on agencies’ Enterprise Risk Management and Internal Controls activities.|
|Who do I talk to?||Chief Risk Officer (CRO)|
|Questions to Consider||In reviewing the agency’s risk profile, are there priority questions that the CRO would recommend be included in the learning agenda? Example: An agency’s risk profile is a living document. The EO and CRO could meet on a quarterly basis to review the risk profile and discuss the impact on priority questions.|
Internal and external stakeholders can offer insights into existing evidence-building activities and ideas to inform an evidence-building roadmap. How do you use this information to create a list of priority questions for your learning agenda?
Depending on your approach to stakeholder engagement, your team may have an extensive list of survey results, public comments, and/or meeting notes with ideas and questions related to research, analysis, and evaluation. At this stage in the learning agenda process, your team may need a process to filter and synthesize stakeholder input to develop a set of priority questions.
If your agency already has a process for developing draft questions, please skip to Step 4 of this section for guidance and a template to present draft questions to senior leadership.
One possible process, as shown in the figure to the right, includes converting stakeholder input into questions, reviewing and refining questions, checking for balance across the questions in required and suggested categories, and presenting the questions to leadership. This process can be tailored to the structure of your core team (Section 3: Engaging Stakeholders), requests from senior leadership (Section 4: Encouraging Leadership Buy-In), and agency needs. Please refer to Worksheet #4 of the Learning Agenda Workb
f your agency opts to have sub-agency learning agendas, this process can be used to determine which sub-agency learning agenda questions will be included in the agencywide learning agenda. Please note that sub-agency learning agendas cannot fulfill the requirement for an agency-wide learning agenda. See the Office of Evaluation Sciences Toolkit Landing Page to download additional information about sub-agency learning agendas.
Convert Stakeholder Input into Questions
In Worksheet #3 of the Learning Agenda Workbook, the Evaluation Officer (EO) and related staff documented stakeholder engagement activities to receive input on potential learning agenda content. No matter how your team chose to engage with internal and external stakeholders – whether it was listening sessions, roundtables, email outreach, internal meetings, agency-wide surveys, or conferences – the suggestions you received on evidence priorities may be in the form of statements. Some practitioners find it useful to convert statements into questions before reviewing them. They also recommend separating multi-part questions and removing duplicative input. For examples of priority questions, refer to Section 2: Types of Learning Questions.
If you formed a core team (see Section 3: Engaging Stakeholders), you may be interested in additional guidance about how to operationalize the question review process for this team. The process below, and guidance in Activity #2 of the Learning Agenda Workbook can be used to form questions that are relevant and meaningful to your agency’s strategy and operations. To follow along with Activity #2 of the Learning Agenda Workbook and the steps below, click here to download the Excel Workbook .
1. Create a Rubric and Vet Questions
What makes a good question for your agency? In Table 5 in Worksheet #4 of the Learning Agenda Workbook, your team can use suggested criteria or create your own criteria to evaluate potential priority questions. Rigorous criteria help ensure that the answer to each question will be used to inform strategic or operational decisions. Setting criteria can help facilitate conversations around what constitutes a good question and what questions are priorities for your agency and leadership. The criteria used by the United States Agency for International Development, as described in the box below, may be a useful reference.
After determining the criteria that are best suited for your agency, you can use Table 6 to score each question and determine whether or not a question should be refined. During this process, stakeholder consultation may be necessary. For example, when evaluating whether a question is “inclusive,” program managers may need to be consulted.
A Tool for Developing Learning Questions from the United States Agency for International Development
This checklist, summarized below, is used by USAID and its implementing partners when developing or reviewing learning questions during monitoring, evaluation, and learning processes
- Useful: Each question should have clear use or applicability to your agency’s work. When determining whether a question is actionable, keep in mind policy considerations, including earmarked funds and agency mandates.
- Focused: Confirm that the question can be understood similarly by multiple people. Define key terms that could have multiple meanings or interpretations. Perform due diligence to make sure the question is still unanswered.
- Feasible: Given the scope of the question, consider whether there are appropriate quantitative and qualitative methods that can be used to answer it, and whether those methods will be considered valid and credible by the users of the learning. Determine the level of effort required and whether the resources required outweighs the anticipated benefits of answering the question.
- Inclusive: For every question asked, there are individuals responsible for carrying out the learning activities to answer the question, and those who we hope will use the answers to inform their work. Collaborate with those stakeholder groups to increase the likelihood that answers to learning questions will be used to adapt and improve programming.
2. Assess Alignment with Strategic Goals and OMB M-19-23 Guidance
OMB M-19-23 Reference
Agencies should identify priority questions that, when answered, will have the biggest impact on agency functioning and performance… Learning agendas should include both short- and long-term questions of interest to the agency, as well as mission-strategic and agencyoperational questions… in a balance that emphasize those of greatest relevance and priority to the agency.
- Is each strategic goal supported by at least one question?
Agencies are not required to, but may align each priority question to a strategic goal. However, agencies are required to list which strategic goals are addressed by the plan.
Do questions support Secretarial or other senior leadership priorities?
Senior leadership are critical stakeholders in learning agenda development. If their priorities are reflected in the set of questions, leadership is more likely to champion the learning agenda.
Do questions overlap with key external stakeholder priorities?
An agency may need external support to answer a priority question in the next four years. Consider which of the field’s priorities align with agency priorities and where overlap may exist to support the group of questions.
Do questions appropriately span sub-agencies?
If there are no questions related to a specific sub-agency, consider revisiting your stakeholder engagement notes or process, as appropriate.
Do questions appropriately span geographic regions covered by our
In the stakeholder engagement process, you may have received feedback from regional offices. Consider how your learning questions reflect the priorities by geographic region.
Do questions reflect consideration of the four components of evidence
noted in OMB M-19-23 (i.e., performance measurement, policy analysis,
program evaluation, and foundational fact-finding)?
During the stakeholder engagement process, you may have documented a variety of evidence-building activities in your agency. Consider how diverse learning activities can help answer priority questions.
If referencing the Capacity Assessment, do the questions align to the areas
of evidence your agency is trying to build capacity in?
Agencies are not required to align the Capacity Assessment and learning agenda. However, if your Capacity Assessment is complete, the EO may have identified specific focus areas for evidence building, such as a program with limited evidence-building activities. Consider how your priority questions will generate evidence to support those focus areas.
3. Categorize Questions
Priority questions are not required to be categorized, but categorization has several benefits. First, the EO and related staff can quickly review sets of potential questions to see where additional questions may be needed (e.g., aligned to a specific strategic goal). Also, the learning agenda is a coordination tool. If questions are categorized, owners could be assigned to a portfolio of similar questions for tracking or accountability purposes for implementation. Lastly, the learning agenda will be shared publicly with stakeholders who will be involved in implementation. Easily understood categories helps stakeholders clearly recognize your agency’s priorities for learning.
There are a variety of methods to categorize your potential priority questions. For example, questions could be mapped to strategic goals, organized by focus areas, or divided by question type. You may find it useful to start by categorizing long-term questions, then deliberate whether a series of short-term questions should be answered first.
As you and your team organize questions, look for crosscutting themes and potential projects. In the leadership presentation template, there is space to include proposed learning activities with the priority questions.
4. Obtain Leadership Approval
As described in Section 4: Encouraging Leadership Buy-In, leadership input and buy-in is critical to the learning agenda process. You may have initiated leadership engagement with a briefing on Evidence Act requirements and documented their needs and priorities. At this stage, a meeting with leadership is an opportunity to show how the set of potential questions is aligned to expectations communicated at the beginning of the process.
Reference your stakeholder engagement matrix for leaders identified for final review and approval of learning questions. Depending on the number of questions to be discussed and individuals attending, time allocated for this meeting may be one to four hours. Consider designating an individual with strong facilitation skills to lead the meeting for a focused discussion that leads to consensus on the priority questions.
Click here to download a PowerPoint template and refer to Activity #3 of the Learning Agenda Workbook to develop a customized presentation for a guided discussion with senior leadership. For more information about how to encourage leadership buy-in, please reference Section 4: Encouraging Leadership Buy-In.
There is no prescribed style or format for the learning agenda. How will you design the learning agenda to be meaningful to your agency?
OMB M-19-23 Reference
At a minimum, agencies should include the following types of content in their learning agenda, recognizing that this content does not necessarily need to be organized according to these headings:
- Which Strategic Goals and Objectives that the Learning Agenda will Address
- Priority Questions to be Answered
- Activities that the agency will engage in to address Priority Questions
- Timing of Learning Agenda Activities
- Potential Data, Tools, Methods, and Analytic Approaches to be Used to Answer Priority Questions
- Anticipated Agency-Specific Challenges and Proposed Solutions to Developing Evidence to Support Agency Priorities
Your agency’s learning agenda is required to be published with the Strategic Plan and Capacity Assessment at least every four years. Like the Strategic Plan, the learning agenda should be written in plain language to make it easy for internal and external stakeholders to understand. While the guidance released by the Office of Management and Budget (OMB) lists specific topics to be included in the learning agenda, as shown in the box to the right, it also emphasizes agencies’ flexibility in designing and structuring their learning agendas (e.g., adding a new section). This toolkit section includes a sample learning agenda question aligned to a strategic goal and a template that categorizes questions by focus area.
Potential Reference Documents
The following documents may be useful to reference while writing your agency’s learning agenda:
- Stakeholder engagement meeting notes: If you completed Worksheet #2 of the Learning Agenda Workbook (available for download at the top of this page), you may already have a repository of ongoing learning activities and potential data, tools, methods, and analytic approaches to answer priority questions.
- Confirm the accuracy of this information with the Chief Data Officer and Statistical Official.
- Section 6: Using Stakeholder Input Excel and/or PowerPoint template: If you conducted an exercise to vet and categorize questions, or received leadership feedback, you may have all priority questions finalized and approved at this stage. If you completed Table 8, you might have all required information related to potential data, tools, methods, and analytical approaches, and anticipated challenges.
- Capacity Assessment results The Capacity Assessment will also be published every four years with the learning agenda. There is no prescribed timing or coordination between the Capacity Assessment and learning agenda. However, some of the requirements may serve as useful inputs, as shown below.
Inventory of activities and programs that are being evaluated
Evidence-building activities to answer priority questions
Assessment of how evidencebuilding activities support agency needs
Timing of learning activities
Assessment of information exchange and follow-through
Challenges and mitigation strategies
Learning Agenda Section Sample
The following learning agenda section sample for the fictitious Department of Societal Betterment can be downloaded at the top of this page. The sample section covers all topics listed in the OMB M-19-23 box above.
The Department of Societal Betterment
Mission:Reduce societal nuisances
Agency Strategic Goal 1: Make Day-to-Day Living in Our Society More Pleasant
Agency Strategic Objective 1.1: Reduce nuisances in public places
Does new signage emphasizing pride in our public places reduce the incidence of littering?
Addressing the priority question(s)
Background. Littering was found to be one of the top three nuisances in our 2019 annual survey, Nuisances that Bug You the Most. Next year we will be implementing an important new initiative that uses signs emphasizing pride
Addressing the question. We will use the phase-in stage of the signage initiative to help us learn about the impact of the signs on littering. In particular, among the 40 sites around the city that have been selected for signage, we will randomly select half (20) to receive the signs first for a six-month period. During that time, there will be 20 sites with signage (program group) and 20 sites without (control group), allowing for a comparison of littering incidence. We will also examine time-trend data at each site to see if littering varies in the months before and after implementation of the signs.
To do these analyses, we will use existing data on the amount of litter collected by our Streets and Parks Departments. We may supplement that information with survey data from users of targeted parks and streets.
One challenge is that there are sometimes special events in particular parks or neighborhoods that can skew the data, so we will need to take those factors into account in the analyses. Another challenge is sample size: the total number of sites (40) is too small to produce very exacting comparisons between treatment and control sites during the phase-in process, but we believe the analyses will nonetheless produce useful information. We plan to work with evaluation experts within the agency, as well as with university partners as needed, to ensure that we have as credible of an evaluation design as possible.
Learning Agenda Template
A fillable template with the headers used below can be found in Worksheet #5 of the Learning Agenda Workbook. As a reminder, this is one of many potential ways to format the learning agenda.
Insert a brief introduction about the agency’s learning agenda aims. Refer to Section 1: Purpose and Benefits for the purpose and potential benefits of a learning agenda.
Insert an overview of the agency’s stakeholder engagement efforts (process and participants), and how internal and external engagement informed the questions included in the learning agenda. Emphasize stakeholder engagement required per the Evidence Act – the public, State and local governments, and representatives of nongovernmental researchers.
Include a brief explanation of how your agency defines “short-term” and “long-term” questions included in the learning agenda.
Focus Area #1 (Optional to organize by focus area)
- Required: Strategic goals addressed in this section.
- Optional: Other priorities addressed in this section.
- Required: Priority questions relevant to that focus area
- Optional: Brief explanatory text that would help readers understand the priority question, including background information or details.
- Strongly Encouraged: If the question is related to a regulatory action, reference the entry in the Unified Agenda of Federal Regulatory and Deregulatory Actions.
Required: Activities that the Agency will Engage in to Address Priority Questions
For each priority question, briefly describe the ways in which your agency will seek to answer that question. This description can be high-level, since the potential methods and analytic approaches will be described below. Consider combining the sections if necessary.
Required: Potential Data, Tools, Methods, and Analytic Approaches to be Used to Answer Priority Questions
This section should expand on the learning agenda activities discussed above by providing more details about potential data sources, tools, methods, and/or approaches that may be used to answer each question. Note the importance of the word “potential” here, as your agency may be just starting the process of identifying how priority questions will be answered. Existing capacity to answer a priority question is not a prerequisite for inclusion. Where capacity is available, agencies are expected to use the most rigorous method that is feasible in answering priority questions.
To demonstrate compliance with existing laws and regulations, consider including a brief explanation of the agency’s data governance related to the potential data sources listed.
Note: The Annual Evaluation Plan will provide additional detail on method(s) to be used.
Required: Timing of Learning Agenda Activities
In this section, briefly describe the timeframe for the activities above. For example, will these activities take place over the next year or span multiple years? If there are several activities associated with one priority question, describe the timeframe for each.
Required: Anticipated Agency-Specific Challenges and Proposed Solutions to Developing Evidence to Support Agency Priorities
Briefly preview one or more of the challenges in undertaking the activities described above. These could be challenges specific to an individual question or broader agency challenges. For each challenge, document a proposed mitigation strategy.
*Repeat the five headers above as needed*
Optional: Learning Agenda Updates
Per the Evidence Act, agencies are required to revisit the learning agenda on a minimum of an annual basis. Include an explanation of the process and stakeholders involved in making meaningful updates to the learning agenda.
See Section 8: Additional Resources to view example learning agendas from other federal agencies. Please note that all learning agendas were developed prior to OMB M-19-23 guidance and may be missing certain required elements.
This section contains additional resources to learn more about learning agendas. While this list is not exhaustive, it provides additional insights from experts in the public and private sectors. Please consider Foundations for Evidence-Based Policymaking Act of 2018 (Evidence Act) requirements and OMB M-19-23 guidance when reviewing tips or examples.
The Office of Management and Budget (OMB) operates an Evidence and Evaluation Community on MAX. This page contains many of the resources below as well as updates, workgroups, and initiatives. Click here to view the community with your MAX account.
Learning Agenda Overview
- Administration for Community Living, ACL Learning Agenda Approach
- Institute of Education Sciences, Developing a Coherent Research Agenda: Lessons from the REL Northeast & Islands Research Agenda Workshops
- Lynette Friedman in Collaboration with Sheila L. Jackson, Learning Agenda Workbook for Food Security and Nutrition Programming
- National Science Foundation, Learning Agendas at NSF
- United States Agency for International Development, Learning Agenda Resources
- United States Agency for International Development, Learning Questions Checklist
- Urban Institute Evidence Toolkit, Learning Agendas
Agency Learning Agendas
Note: These example learning agendas were created prior to the passage of the Evidence Act; therefore, they can be informative but are not fully compliant with the requirements of the Evidence Act and OMB M-19-23.
- Housing and Urban Development, Research Roadmap
- Small Business Administration, Enterprise Learning Agenda
- United States Agency for International Development, Self-Reliance Learning Agenda
- Evidence-Based Policymaking Collaborative, Engaging Stakeholders in Learning Agenda Development
- United Nations Economic and Social Commission for Asia and the Pacific, Effective Stakeholder Engagement for the 2030 Agenda
- United States Agency for International Development, Collaborating, Learning, and Adapting Toolkit: Engaging Stakeholders
- Urban Institute, Engaging Stakeholders in Learning Agenda Development
Building a Culture of Continuous Learning and Improvement
- Urban Institute, Building Evidence Culture and Capacity in Federal Agencies
- Grantmakers for Effective Organizations, Learning in Philanthropy Guidebook
- PropelNext, Becoming a Learning Organization
Evidence-Based Policymaking Trainings
Note: This list is not exhaustive, but rather is meant to be representative of the various mediums and topics available to agency leadership and staff.
- The American Evaluation Association publishes a number of Training and Professional Development tools such as eStudies, evaluation events, presentations, and coffee break webinar series. Topics include evaluation-related methodologies, evaluation tools, business and technical skills for evaluators, and methods for communicating evaluation results to specific audiences.
- The William T. Grant Foundation has compiled syllabi and training manuals focused on improving the use of research and evidence in public policy settings. The training manuals address topics related to using research evidence, developing relationships with researchers, formulating research questions to inform policy decisions, accessing research evidence, and working with a range of evidence.
- The Office of Personnel Management (OPM) hosts monthly sessions via the Federal Employee Development (FED) Evalu-Add Learning Series that cover topics such as theory of change, logic models, learning design, and decision-making. These sessions aim to showcase effective and innovative evaluation practices and tools that will increase agencies’ evaluation abilities and help improve training program management. The sessions are intended for federal learning and development professionals, especially those with program evaluation responsibilities.
- The USAID Learning Lab Collaborating, Learning, and Adapting Toolkit curates the “growing set of curated tools and resources on Collaborating, Learning and Adapting to help you plan and implement Program Cycle activities more effectively.” Please note, this toolkit is written specifically with USAID staff members and USAID program cycles in mind, but may still contain useful tips for your agency.
- The Environmental Protection Agency Logic Model Training is a web-based online training course that explains logic model concepts, elements of logic models, outputs and outcomes, and case study examples.
- The Corporation for National and Community Service Evaluation Core Curriculum courses are designed to help increase program capacity to evaluate interventions and therefore improve services to serve more people, better. Topics include planning, implementation, reporting, and using results for action and improvement.
- The Center for Disease Control and Prevention evaluation resources contain self-study guides on program evaluation, frameworks, evaluation workbooks and tools, and tips for planning and conducting strong evaluations.
- The Institute of Education Sciences Logic Model Workshop Toolkit explains the purpose, elements, and steps for developing and using logic models for program evaluation. Topics include helping users identify indicators of success and developing strategies to determine the right evaluation design for their programs or policies.
- J-PAL Executive Training Evaluating Social Programs is a five-day program on evaluation social programs. The goal of this course is to increase understanding of randomized evaluations and step-by-step training for conducting one’s own evaluation. Many of the topics, including measuring outcomes and managing threats to validity, are relevant for other methodologies. This training also includes a video lecture.
- The Evaluator’s Institute (TEI) offers professional training and certifications to build capacity in evaluation foundations; theory, design, and methods; approaches and techniques; strategies and capacity; and analytic approaches. TEI offers programs and events, courses, and certifications in evaluation topics.