During recent years, political science scholarship has witnessed a considerable development of automated content analysis techniques, with Wordscores and Wordfish as the two most notable contributions. These two methods of analysis have conventionally been used to study political parties’ manifestos and legislative speeches. Yet recently, one of them, namely Wordfish, was suggested as a suitable instrument for estimating interest groups’ policy influence in the context of EU policymaking, and it was applied to analyze interest groups’ policy position documents submitted as part of the European Commission’s open consultations (see Klüver 2009). While agreeing that methodological advancements made in the field of party politics can make a substantive contribution to the development of interest group research, the present paper argues and illustrates empirically that current applications of Wordfish to the study of EU interest groups raise a set of problematic theoretical and methodological issues that need to be more carefully considered and addressed by scholars of EU lobbying.
Using quantitative analyses and an alternative method of content analysis, the study provides the first systematic, critical evaluation of both theoretical and methodological shortcomings marking current applications of quantitative text analysis to the study of EU interest groups’ policy influence. The consequences of important methodological trade-offs required by the use of this method, such as for example discarding important substantive information as well as observation points, or artificially enhancing the length of analyzed documents so as to fit software requirements, are just some of the aspects considered in the analysis. The paper contributes to (1) the literature discussing valid measurements of interest groups’ policy influence in the EU and other systems of governance, and (2) to the more recent scholarship on the use of automated content analysis in political science across different fields of research.
