Tackling Falsehoods with Generative AI: A Systematic Cross-Topic Examination of Chatbot Capacity to Detect Veracity of Political Information

The viral launch of ChatGPT and the subsequent chatbots (e.g. Bing CoPilot) has triggered a plethora of studies looking into its various applications: in education, marketing, law, and politics, to name a few. Powered by Large Language Models (LLMs), a category of AI models capable of generating and processing text (Naveed et al., 2023), chatbots have the potential to redefine the role of AI in politics. Unlike earlier non-generative forms of AI, such as search engine algorithms, LLMs do not only retrieve information but can also interpret semantic nuance of content they generate or interact with. While they have the potential to create false information or facilitate government censorship in authoritarian states (Urman & Makhortykh, 2023), LLMs also present new opportunities for content analysis, including the identification of misinformation. This application is garnering increasing attention in political communication studies (e.g., Hoes et al., 2023; Zhang & Gao, 2023). However, the automated assessment of information truthfulness remains a complex challenge. This article presents the first large-scale cross-topic multilingual analysis of the ability of the large language model (LLM)-based chatbot ChatGPT 4.0 to detect veracity of political information. We use AI auditing methodology (Falco et al., 2021; Kuznetsova et al. 2023) to investigate how chatbots evaluate the veracity of true, false, and borderline statements on 6 common misinformation topics: COVID-19, immigration, race, the Holocaust, climate change, LGBTQ+ as well as the ongoing conflicts: the war in Ukraine and the Israeli and Palestine conflict. We compare how ChatGPT 4.0 performs in high- and low-resource languages by using prompts in English, German, Spanish, French, Italian, Polish, Hindi, Portuguese, Arabic, Russian, and Ukrainian. Furthermore, we explore chatbots’ ability to deal with political communication concepts of disinformation, misinformation, and conspiracy theory, using definition-oriented prompts and systematically testing the presence of source bias by attributing specific claims to various political and social actors. With the help of OpenAI API, we examine a structured corpus of prompts comprising known and fact-checked false claims as well as made-up statements, to examine LLMs ability to assess veracity of politics-related statements. This research highlights the potential of chatbots in tackling different forms of false information in online environments but also points to the substantial variation in terms of how such potential is realised due to specific factors (e.g. language of the prompt or the information topic).