Difficult Wordings in VAAs. Predicting No-Opinion and Middle-Response Answers from Question Characteristics

Baka et al. (2012) investigated the reasons for VAA respondents to choose the middle-option or “no opinion” in the Greek VAA Helpmevote. Van Outersterp et al. (2016) studied the same for the Dutch Stemwijzer. Although the labeling of the answering options in both VAAs studied differed, they reached similar conclusions: the middle-response option and the non-substantive option are both used to express indifference, ambivalence, and a lack of knowledge. Mixing these response options' meaning is problematic for VAAs, as the resulting voting advice is less accurate. Kamoen and Holleman (2017) further investigated the relationship between wording characteristics and middle-category answers and non-substantive answers. Based on a qualitative think-aloud study with 40 respondents filling out the VAA Kieskompas and 20 respondents filling out Stemwijzer for the 2014 municipal elections, the authors found several wording characteristics that systematically led to comprehension problems. Two main categories of comprehension problems were distinguished: semantic problems (a lack of understanding of the meaning of words or concepts) and pragmatic problems (a difficulty to grasp the context, e.g., a current state of affairs). Departing from these qualitative findings they studied the use of the middle-option and of the non-substantive answering option (“no opinion”) quantitatively, by predicting the occurrence of middle-response and no-opinion answers in VAA Kieskompas based on question characteristics. This showed that it was possible to predict problematic answers based on some of the question characteristics: questions containing semantic difficulties (e.g., tax names) were related to a significant increase of non-substantive responses, whereas the use of so-called status quo triggers (pragmatic problems) was related to an increase in middle-category responses only. Research aim: Current study examines to what extent the conclusions drawn in Kamoen and Holleman (2017) based on Kieskompas-data can be replicated for the VAA brand Stemwijzer. Stemwijzer differs in several key aspects from Kieskompas, e.g., it uses a three-point response scale instead of a five-point scale, and applies distinct labels to both the middle option and the non-substantial option. Method: The previous Kieskompas analysis was conducted on all answers provided by 357,858 users who accessed one of the 34 municipal VAAs of Kieskompas for the Municipal elections in 2014 in the Netherlands. Current analysis was conducted on all answers obtained Stemwijzer during the Municipal Elections of 2018, for 44 municipalities in the Netherlands. In each VAA, between 1,321 and 78,855 VAA users provided an answer to 25-30 questions, resulting in about 9.1 million data points. Results: In line with earlier Kieskompas findings, the choice for a middle-category answer or non-substantive answer is not random: question characteristics associated with semantic meaning problems often result in no-opinion answers, whereas characteristics triggering pragmatic meaning problems are more frequently related to middle-category responses. This suggests that although both the middle response and the non-substantive response options are used to express comprehension problems, there is–at the same time- a difference in their perceived meaning. We will discuss various ways to avoid resulting validity problems.