Introducing the CAP BABEL MACHINE: A state-of-the-art AI solution for automated coding of policy agendas

The article presents the CAP BABEL MACHINE, a natural language processing tool developed by poltextlab.com in Hungary, which was designed to simplify and speed up research projects for comparative policy agendas scholars. The methodology behind the tool is based on the coding systems of the Comparative Agendas Project (CAP) and state-of-the-art large language models (LLMs – such as BERT and RoBERTa) to provide an efficient, reliable, and inexpensive alternative to manual coding. The pipeline allows for the self-service upload of data, which, after an initial, automated quality check, triggers the respective fine-tuned CAP classification model. The service also provides the option of self-service evaluation by also uploading human-coded test data. In the current development phase, model accuracy for English, Hungarian, Spanish and Polish stands between 0.74 and 0.83, with further improvements going live on an ongoing basis.