A multiverse analysis of moral rhetoric in multilingual textual corpora

Extracting moral information from public discourse is a critical step for understanding how moral rhetoric unfolds at a global scale. However, the contextual and cultural embeddedness of morality presents numerous challenges for the valid measurement and analysis of moral cues in multilingual textual corpora (Hopp & Weber, 2020). The majority of past work has utilized the deliberately constructed, expert-driven moral foundations dictionary (MFD; Graham, Haidt, & Nosek, 2009), its computationally-enlarged successor MFD2.0 (Frimer et al., 2017), and the crowd-sourced, extended moral foundations dictionary (eMFD; Hopp et al., 2020) for detecting morally relevant words in English textual corpora. Furthermore, extant analyses have either counted the presence of moral terms in texts or computed the semantic similarity between texts and selected moral keywords (Garten et al., 2018). Analogously, researchers have translated dictionaries (Bos & Minihold, 2022) or textual corpora (Malik et al., 2021) to study moral rhetoric in non-English sources. In view of this emerging work, researchers now face a multitude of decisions when aiming to measure moral rhetoric in multilingual texts. In this paper, we propose a multiverse analysis to examine how existing dictionaries, scoring techniques, and translation decisions impact the results of a computational moral content analysis on multilingual texts. As a test bed, we use the openly available corpus of the Comparative Manifesto Project (CMP; Krause et al., 2018), which provides more than 1,800 machine readable, native language party manifestos from 40 different countries. Because natural language processing is prone to questionable research practices (QRP, Bakker et al., 2022) in the form of opaque, black-box text processing algorithms, we aim for maximum transparency by documenting all our analysis steps in open, reproducible Python notebooks. To evaluate our multiverse analysis, we will assess the statistical properties of the extracted moral signal (e.g., mean, variance, skew); the extent to which the computed moral signal converges and diverges; and how well ideological differences in the moral framing of policy issues can be crystallized. Taken together, we hope that our paper will provide researchers with a transparent and instructive roadmap that details how methodological decisions impact the validity and results of computational moral content analysis (Hopp & Weber, 2020) and contribute to growing pains for analyzing multilingual textual corpora (Lind, Eberl, Heidenreich, Boomgaarden, 2019).