The randomness factor: How LLM-based chatbots reiterate and counter Russian disinformation about the war in Ukraine

The rise of large language models (LLMs) leads to major transformations in how individuals engage with political information. The capacity of LLMs to recognize human input and generate textual outputs facilitates the integration of conversational interfaces in platform architectures. These interfaces enable exchanges between platform users and LLM-based chatbots that has major implications for information-seeking behavior considering that individuals tend to be more accepting of (counter-attitudinal) information acquired via chatbots due to their anthropomorphism. These implications are particularly important regarding information about issues that are often targeted by disinformation: in this context, LLM-based chatbots can enable new possibilities for debunking false claims but also amplifying their distribution. The potential impact of LLMs on information consumption is particularly relevant for search engines (SEs). Recognized as key information gatekeepers, SEs such as Google currently experiment with the use of LLM-based chatbots for improving their services. However, the integration of LLMs and SEs raises concerns due to both types of information systems often being non-transparent and subjected to different forms of bias. These concerns are particularly pronounced due to the possibility of LLM bias contributing to the promotion of false claims, particularly in the interests of authoritarian regimes. However, there is still little understanding of how systematic is the generation of misleading outputs by SEs-integrated chatbots, in particular considering their stochastic nature, and to what degree it is influenced by different factors (e.g. input language). In the light of the above-mentioned gaps, we aim to make the three contributions in this paper. First, we examine the accuracy of SEs-integrated chatbot outputs regarding the ongoing war in Ukraine which is characterized by the large number of false claims made by the Russian authorities to justify its aggression and mislead the public opinion in different parts of the world. Second, we investigate how the accuracy of outputs varies across different languages in which prompts are written. Third, we analyze how the accuracy of outputs is influenced by the stochasticity of the LLMs powering them. To make these contributions, we conduct AI audits of three SEs-integrated chatbots - Bard, Co-Pilot, and Perplexity AI - using 28 prompts. The selection of chatbots was attributed to our interest in comparing the performance of chatbots designed by major tech giants (Google and Microsoft) with a startup-made chatbot. All prompts were related to common disinformation claims such as Ukraine being ruled by Nazis or developing biological weapons to attack Russia. Prompts were phrased in the form of polar (yes/no) or open-ended questions and translated in English, Ukrainian, and Russian. To examine the impact of stochasticity, four assistants generated chatbot outputs around the same time with the same set of prompts. The resulting 1,008 responses were analyzed according to a custom-made codebook evaluating whether the output (1) matches the expert baseline; (2) reports the Russian interpretation of the issue; (3) refers to the Russian position as disinformation/propaganda. Preliminary findings indicate that chatbots in some cases amplify Russian disinformation and the quality of information is affected by stochastic factors.