Promises and pitfalls of using large language models to identify actor stances in political discourse

A recurring requirement of natural language processing applications in policy process research is stance detection in political discourse. In a typical use case, this involves the identification of the qualitative stance (such as support or oppose) of a specific policy actor (such as an interest group) regarding a specific statement (such as a policy belief) in a given textual fragment (such as a newspaper paragraph). Automated text analysis relying on this use case allows to scale policy process research applications in eg. analyzing discourse networks or coalition formation and change over time. Formulated in this form, this problem is however still non-trivial, even after recent mainstreaming of substantial advances in NLP techniques. The key problem is that existing stance detection approaches perform well and are mostly geared toward classifying statements as unary relations, eg. made from a singular speaker's point of view in social media, and are often statement-agnostic. Treating stance classification as a binary relation, thus classifying statement-specific stances of specific actors within paragraphs where multiple actors might be written about in the third person, which is common if analyzing media data, is still a task where many research projects within policy process research will struggle to achieve adequate performance. On the face of it, recent advances in large language models (LLMs) promise a remedy. LLMs are incredibly general-purpose language processing tools and fine-tuning them for stance detection is theoretically a way to boost the availability of stance detection for policy process research projects. Here, we develop a robust application of self-hosted LLMs (LeoLM variants) orchestrated using the guidance framework to classify stances of organizations in Swiss newspaper article paragraphs regarding a set of policy beliefs related to sustainable urban transport topics. We evaluate the performance of our LLM-based application in comparison to manually labeling a representative sample of texts, a standard rule-based and a transformer-based text classification architecture based on classification performance but also with regard to ease of use, inference costs and explainability.