Is a single model enough? Lessons learned from systematically comparing automated classifications of populist radical right content in German

The rise of populist radical right (PRR) ideas stresses the importance of understanding how individuals are exposed to and engage with PRR content in high-choice information environments. However, this task is complicated by the multitude of channels via which such exposure can take place. This prompts the need for developing automated approaches for identifying PRR content. In this paper, we share insights from our experience of developing automated classifiers for differentiating between PRR and non-PRR textual content in the German language. By training and comparing 66 dictionary-, supervised machine learning-, deep learning-, and transformer-based classification models, we offer a systematic comparison of their performance on three validation sets of PRR textual items and examine the impact of different pre-processing steps (e.g., stemming and lemmatization) on models’ performance. We also discuss the use of synthetic models (i.e., combining individual classification models in the ensemble form) for PRR classification based on a comparison of 396 model combinations. Our findings demonstrate that transformer- and supervised machine learning-based models show the best performance on average among the individual models and it can further be improved using synthetic models which combine supervised machine learning- and dictionary-based approaches.