You Get How You Ask For: Survey Mode Effects in the European Social Survey (ESS) Round 12

How a survey is administered can significantly influence how respondents answer. However, opportunities to compare the effects of different survey modes across a wide range of variables and contexts in an experimental setting are rare. This paper exploits the unique design of Round 12 of the European Social Survey (ESS), which implements a mixed-mode approach: half of the interviews are conducted face-to-face, while the other half use self-completion (web or paper), with interview mode randomly assigned to respondents. We expect that the presence of an interviewer will influence responses respondents give. Our analysis includes over 15,000 respondents from 13 countries, yielding more than 1.5 million responses on 96 items with ordered response scales covering 20 different themes. The results show that, on average, responses in the self-completion mode diverge significantly from those in the face-to-face mode. However, the magnitude of these differences varies considerably across themes, and the themes most affected by mode effects differ by country. These findings highlight that the choice of survey mode can have a substantial impact on response patterns, with important implications for the comparability of data across countries and over time, as well as for the interpretation of analyses based on survey data.