Auditor General Slams Ottawa’s $18 Million CRA Chatbot ‘Charlie’

The Canada Revenue Agency is facing renewed scrutiny after the Auditor General found its taxpayer-funded chatbot frequently delivers incorrect answers, despite costing Canadians more than $18 million to develop and operate.

The findings were reported by the National Post and stem from an October audit of CRA call centres, which also examined the agency’s digital self-service tools. One of those tools, a chatbot named Charlie, remains live on multiple CRA web pages.

Charlie was introduced in February 2020 by then national revenue minister Diane Lebouthillier, who said it would help Canadians better navigate the tax system. At the time, the CRA warned the chatbot was still “learning.” Five years later, the Auditor General says that learning curve has been expensive and ineffective.

“Charlie’s responses tended to be brief, offering limited context and minimal additional information,” Auditor General Karen Hogan wrote in her report.

Her office tested the chatbot using six tax-related questions. The results were stark. “We found that Charlie provided accurate answers in only 2 out of the 6 questions we asked it, while the other public web-based conversational artificial intelligence tools answered 5 out of 6 questions accurately.”

The comparison raised concerns in Parliament, particularly because Charlie is a government-built tool designed specifically for tax questions. Liberal MP Anthony Housefather pressed Hogan on the discrepancy, asking, “how is it that the government’s tool specialized for the CRA is worse than the average tool that anyone else can access anywhere?”

Parliamentary documents show the CRA has spent more than $18 million on Charlie since the 2018–2019 fiscal year, with most of the spending occurring after its broad rollout in 2021–2022. Salaries accounted for $13.67 million of that total, another $3.21 million went to IT consultants, and the figures do not include employee benefits or travel costs.

The new GenAI beta chatbot so far looks to be better than its initial rollout, as it was able to answer a question on what personal income tax tiers are, plus cite sources. But is this worth $18 million?

The CRA says Canadians have used the chatbot heavily, with more than 7 million conversations and over 18 million questions since its launch. Internal testing at one point showed Charlie met a 70 per cent accuracy threshold, meaning nearly one-third of responses were still incorrect.

In November, the agency upgraded Charlie to a generative AI version, similar to tools like ChatGPT. The CRA claims pre-release testing showed an accuracy rate of “approximately 90%,” but acknowledged it cannot confirm real-world accuracy without reviewing every interaction.

As of the end of the 2024–25 fiscal year (March 31, 2025), the Canada Revenue Agency (CRA) reported 53,585 full-time equivalent employees across its core responsibilities and internal services.

The amount of taxpayer funds spent on the chatbot pales in comparison to the $60 million spent on the ArriveCAN app, which is likely saying, “hold my beer, Charlie”.

Have you used Charlie the chatbot on the CRA website? It seems using something like Google’s Gemini or ChatGPT will give you a better answer and faster.