Evaluating Generative Artificial Intelligence (GenAI) chat tools

Agency priority

GSA is a leader in modernizing and streamlining technology across government, including promoting responsible AI innovation in support of the Administration’s Executive Order to accelerate federal AI use. To support AI innovation and enhance government efficiency, GSA developed an internal GenAI chatbot (GSA chat) and was interested in learning about employees’ experience using chat to improve the future product.

What we evaluated

We partnered with GSA’s Office of Information Technology, Voice of the Customer team, and Federal Acquisition Service’s Office of Strategy and Innovation to evaluate GSA’s use of GenAI tools, with particular interest in GSA chat.

This evaluation had three primary goals:

1. Understand GenAI usage, particularly GSA chat¹,
2. Catalog the barriers users face to engaging with GSA chat, and
3. Understand user interest in additional training and/or support.

We analyzed three data sources for this evaluation. First, user logs from GSA chat’s database (telemetry data) from the first five weeks post-launch (March 21-April 25, 2025) were analyzed to understand how GSA employees engaged with GSA chat (e.g., prompt writing, AI model selection). Second, data from an online survey was analyzed to assess GSA employee attitudes and perceptions of GenAI tools, including GSA chat.² Third, we conducted six semi-structured follow-up interviews with survey respondents to better understand user needs and barriers.

What we learned

We found that GSA chat was used by 3,959 employees within its initial five weeks, representing approximately 35% of GSA employees.³ Of the 281 survey respondents, those who frequently used GSA chat were more likely to participate in the survey. Key findings are described below.

1. GSA chat usage. Most users only prompted GSA chat a few times, with a median of six prompts, and 82% of users used only the default AI model set by GSA chat. Only 16% of users provided feedback on GSA chat’s responses. Among survey respondents who used GSA chat, the most frequently cited benefit was efficiency and time savings, while the primary use cases reported were drafting/editing text, summarization, and research. More frequent GenAI use was associated with the belief that GSA chat would improve work productivity.
2. Barriers to increasing usage. The top selected barriers to increased GSA chat use were inaccurate content (33%) and poor output quality (33%). The top selected drawback was that GSA chat was not integrated into existing workflows (52%).
3. Training or additional support. Respondents were most interested in training and additional support on practical applications of GenAI tools (such as coding) and guidelines for GSA chat usage.

Types of GSA chat users: To gain a clearer understanding of engagement with GSA chat, we categorized survey respondents based on their frequency of use. We then examined variations in their survey responses. Two overarching groups were created: those who had never used GSA chat (“Never” users) and those who had used GSA chat (GSA chat users).

• “Never” users. Although telemetry data indicated about 65% of GSA employees never used GSA chat during the evaluation period, only 26% (n=72) of survey respondents reported not using GSA chat. We identified two distinct types of “Never” users in the survey:
  1. GenAI-avoidant users, who did not try any GenAI tools during the study period (n=33).
  2. GSA chat-avoidant users, who did not try GSA chat though used other AI tools (n=39).

There were key differences in perceived benefits, barriers, drawbacks, and training interests among the two “Never” user groups (Table 1).

Table 1. The most frequently selected benefits, barriers, drawbacks, and training interests among two categories of “Never” user groups

• GSA chat users. We divided GSA employees who used GSA chat into three user groups:
  1. “Tried it” users had tried GSA chat at least once (24% of GSA, 38% of survey respondents).
  2. “Exploratory” users had used GSA chat weekly (9% of GSA, 22% of respondents).
  3. “Power” users who use GSA chat at least daily (2% of GSA, 14% of respondents). There were key differences in perceived benefits, barriers, drawbacks, and training interests among the three user groups (Table 2).

Table 2. The most frequently selected benefits, barriers, drawbacks, and training interests for three GSA chat user groups

Applying the findings

Employee experiences and training needs differed depending on their level of GenAI use, which has implications for increasing AI adoption and use. Although there are some similarities across groups with varied levels of usage, these groups face different barriers to increasing AI use and have different training and support needs. To maximize the potential of GSA chat and encourage broader GenAI use, we recommend the actions below.

Reduce barriers to AI adoption:

• Gather additional input from employees who never use GenAI chat tools to better understand how to increase adoption among this group.
• Implement consistent ways of tracking all forms of GenAI usage to better understand AI adoption across GSA. Most employees are using GSA chat in conjunction with other GenAI tools.
• Develop GSA chat as part of an ecosystem of other AI and productivity tools to address the top drawback that GSA chat was not integrated into existing workflows.
• Prioritize development of new product features based on needs of lower-usage groups to increase GenAI adoption. Maintain human oversight of GenAI tools to monitor and improve accuracy and output quality, top barriers to increased use.

Provide additional training and support:

• Provide concrete use cases for specific job functions to help employees understand how GSA chat can be useful in their jobs.
• Continue to clarify what people are permitted to do in GSA chat.
• Target training for users who have prior, but not extensive, experience with the tool. “Exploratory” users were most interested in trainings, so those will likely be most effective for that group.
• Ensure availability of up-to-date GenAI trainings that include guidelines for GSA chat usage, practical applications, - and safeguards for inaccurate content and output quality. These could be stand-alone trainings and/or incorporated into existing mandatory trainings (e.g., data literacy).

Notes:

1. Includes any GenAI tools available during the evaluation period (GSA chat, Gemini chat, Gemini Workspace, ChatGPT).
2. Online survey available to all GSA employees from May 5 - June 2, 2025.
3. The estimated number of GSA employees was 11,452, calculated using 13,113 employees as of April 30, 2025 (D2D “GSA Organization” data) minus 1,661 who had accepted the Deferred Resignation Program and were excluded from calculations.

This formative evaluation did not have an analysis plan.