Why Microsoft's Copilot AI falsely accused a court reporter of the crimes he reported on

When German journalist Martin Bernklau entered his name and location into Microsoft's Copilot to see how his articles would be received by the chatbot, he was appalled by the responses.

Copilot's findings had revealed that Bernklau was an escapee from a mental institution, a convicted child abuser, and a con artist who targeted widowers. Bernklau had worked as a court reporter for years, and the artificial intelligence (AI) chatbot had falsely blamed him for the crimes he had reported on.

The allegations against Bernklau are, of course, false and are examples of generative AI “hallucinations,” which are inaccurate or nonsensical responses to a user-provided prompt that are alarmingly common with this technology. Anyone attempting to use AI should always proceed with great caution, as information from such systems must be validated and verified by humans before it can be trusted.

But why did Copilot hallucinate these horrible and false accusations?

Copilot and other generative AI systems like ChatGPT and Google Gemini are large language models (LLMs). The underlying information processing system in LLMs is known as a “deep learning neural network,” which uses a large amount of human language to “train” its algorithm.

From the training data, the algorithm learns the statistical relationships between different words and how likely it is that certain words appear together in a text. This allows the LLM to predict the most likely answer based on calculated probabilities. LLMs do not have any actual knowledge.

The data used to train Copilot and other LLMs is extensive. While the exact details of the size and composition of the Copilot or ChatGPT corpora are not publicly disclosed, Copilot includes the entire ChatGPT corpus as well as Microsoft's own specific add-on articles. ChatGPT4's predecessors – ChatGPT3 and 3.5 – famously used “hundreds of billions of words”.

Copilot is based on ChatGPT4, which uses a “larger” corpus than ChatGPT3 or 3.5. Although we don't know exactly how many words that is, the jumps between different versions of ChatGPT tend to be orders of magnitude larger. We also know that the corpus includes books, academic journals, and newspaper articles. And therein lies the reason why Copilot hallucinated that Bernklau was responsible for heinous crimes.

Bernklau had regularly reported on criminal trials of abuse, violence and fraud, which were published in national and international newspapers. His articles are likely to have been included in the language corpus, which uses specific words referring to the nature of the cases.

Because Bernklau worked as a court reporter for years, the most likely words associated with his name when Copilot is asked about him relate to the crimes he covered as a reporter. This is not the only case of this type and we will likely see more of them in the years to come.

In 2023, US talk show host Mark Walters successfully sued OpenAI, the company that owns ChatGPT. Walters hosts a show called Armed American Radio, which discusses and promotes gun ownership rights in the US.

The LLM had hallucinated that Walters had been sued for fraud and embezzlement of funds by the Second Amendment Foundation (SAF), a US organization that advocates for gun rights. This happened after a journalist asked ChatGPT about a real and ongoing court case involving the SAF and the Attorney General of Washington State.

Walters had never worked for the SAF and was not involved in any way in the case between the SAF and the state of Washington. However, since the foundation has similar goals to Walters' show, it is safe to assume that the textual content of the speech corpus built a statistical correlation between Walters and the SAF that caused the hallucination.

corrections

It is nearly impossible to fix these problems across the entire language corpus. Every single article, sentence, and word in the corpus would need to be closely examined to identify and remove biased language. Given the size of the dataset, this is impractical.

The hallucinations that people falsely associate with crimes, as in the Bernklau case, are even more difficult to detect and treat. To permanently fix the problem, copilot Bernklaus would have to remove his name as the author of the articles to break the connection.

Read more: AI can now attend a meeting and write code for you – here's why you should be careful

To address the issue, Microsoft has developed an automated response that appears when a user alerts Copilot to Bernklau's case. The response describes the hallucination in detail and makes it clear that Bernklau is not guilty of any of the allegations. Microsoft has stated that it continually incorporates user feedback and issues updates to improve its responses and provide a positive experience.

There are probably many more similar examples yet to be discovered. It is impractical to cover every single problem. Hallucinations are an inevitable byproduct of the way the underlying LLM algorithm works.

As users of these systems, we can only determine whether the results are trustworthy by checking their validity using established methods. This might include finding three independent sources that agree with the LLM's statements before accepting the results as correct, as my own research has shown.

For the companies that own these tools, such as Microsoft or OpenAI, there is no truly proactive strategy to avoid these problems. All they can really do is react to the discovery of similar hallucinations.