Algorithmically-Assisted Argument Mining in Online Policy Debates: A Comprehensive Analysis of Text Corpora Using LLMs

Argument and debate form cornerstones of public sphere. The study of online civic participation in the public sphere, debates and argumentation is becoming one of the most important challenges that political science, particularly the study of public policy, will face in the near future. Recognising and understanding arguments are central to policymaking and policy analysis. Many researchers (i.e. Landemore) consider the deliberative model—shaped by the works of Habermas and Rawls, and further developed by J. Cohen, J. Fishkin, and J. Elster—as the most appropriate normative framework for analyzing debates on the Internet. To comprehensively explore the dynamics of online policy debates, we aim to expand the ability to identify the key argumentation patterns within them. Argument Mining (AM) is a multidisciplinary research field that encompasses diverse areas such as computer science, logic and philosophy, linguistics and rhetoric, and only recently gaining more attention in social sciences. In AM, several artificial intelligence-based techniques are applied, including natural language processing (NLP), semantic and logical analysis, and deep learning. AM methods can enhance the efficiency of public opinion analysis by, for example, identifying and interpreting argumentative feedback or classifying statements as either supporting or opposing a particular decision. The effective use of this technique requires large text corpora that aggregate opinions expressed in public debates on various issues, such as migration, international conflicts, or health policy. These corpora can subsequently be used to train artificial intelligence models capable of identifying arguments in debates, including those on social media. Our research involves a comparative analysis of three widely recognized corpora used in Argument Mining (AM), each developed by a distinct research team. The datasets analyzed are: 1. The US2016 corpus, which includes 12,392 arguments presented during the 2016 U.S. presidential campaign. 2. The UKP corpus, which contains 25,492 arguments spanning eight controversial topics such as capital punishment, abortion, and cloning. 3. The Args.me corpus, which aggregates 48,798 arguments collected from four debate portals: Debatewise, IDebate.org, Debatepedia, and Debate.org. The topics particularly focus on: international affairs, armed conflicts, nuclear energy, censorship, and health policy. These datasets have gained recognition in recent argument mining research; however, they have not been comparatively analyzed until now. In our study, we conducted argument classification tests using state-of-the-art Large Language Models (LLMs), including LLAMA and GPT-4. Our results highlight a significant proportion of misclassifications made by human annotators and a consistent error rate in classifications performed by LLMs. These findings indicate that existing corpora still contain a considerable number of errors, making them less than ideal as standalone tools for identifying political arguments on social media. However, we demonstrate significant potential for refining text corpora and improving the efficiency of argument mining (AM) processes. Our findings contribute to ongoing discussions about the reliability of AI in real-world applications, particularly in the context of understanding online political debates. They underscore the importance of continued research to enhance model performance and address existing limitations.