Survey experiments and the shadow of professional respondents’ real-life experience: A split-ballot experiment on local politicians following local government amalgamation

Survey vignette experiments have a fast-growing popularity in political science and local government studies. Scholars describe hypothetical scenarios and ask respondents to answer questions about, for example, what a local politician should do when facing a local government merger. Respondents are randomly assigned to treatment groups that receive different versions of the vignette. Researchers will often want to study scenarios that, for example, undergrad students cannot comprehend; they therefore want real-life professionals as respondents to ensure a high ecological validity of the experiment. This paper studies a methodological problem for survey vignette experiments, namely that professional respondents can have real-life experiences and preestablished attitudes that prevent them from treating the scenario as a hypothetical. As a result, their processing of facts in the survey vignette experiment becomes biased. A solution is to control for the respondents’ observed experience in analysing the material. This solution is not available if the researcher lacks relevant information about the respondents. Also, controls are often crude and can overlap with theoretically relevant variables. And in any experiments with participants randomly assigned to treatment groups, using controls is a contested practice. We ask how results from experiments are affected by professional respondents’ preestablished attitudes about- and past experience with a scenario described in a vignette. We use a split-ballot experiment to check if such biases are conditioned by whether the vignette is preceded in a survey by questions that evoke the respondent’s memories of their own experience with a similar situation. The research design thus utilizes question order bias, that is, respondents answering questions differently based on where they appear in a given questionnaire. The study context is a survey of Norwegian local government councillors. The survey was in the field just as many local governments had been involved in amalgamations. The survey contained several sections, with one containing questions about the councillor’s views on and personal experience with local government amalgamations and another containing a survey vignette experiment depicting a hypothetical scenario about whether one should overspend on local goods before a merger. In a split-ballot experiment, the respondents were randomly split in two groups with one group receiving a version of the survey with the vignette first and questions about their views on and experience with municipal amalgamation second; the other group received a version with the reversed order. The results show, first, that the likelihood that councillors support pre-merger overspending is positively affected by their having recent amalgamation experience and a negative preestablished attitude to amalgamations. Second, the results are affected by the order of questions in a survey. The councillors attitude bias is significantly stronger if the vignette is located after survey items that remind respondents of their attitude to amalgamation, compared to the reverse order.