How to Estimate the Policy Preferences of Party Supporters: Disaggregating VAA Data versus Modeling Survey Responses

Good estimates of the policy preferences of political party supporters are essential for a considerable number of theoretical and practical concerns in political science. However such estimates are not readily available, because surveys are not designed to produce subgroup estimates of public opinion at the level of different political parties and traditional surveys rarely ask questions about specific policies. In this context, voting advice applications (VAA) can provide a valuable source of data. VAAs ask specific policy questions and have many more respondents than traditional surveys. However, since VAA users are not a random sample of the total population, it is unclear to what extent VAA data can provide valid estimates of public preferences. To address this concern, we compare two approaches for recovering subgroup preferences at the party level. The first approach is disaggregation of VAA responses. The second approach is multilevel modeling with poststratification (MRP) of traditional survey data. MRP is a technique for producing subgroup estimates from data representative at a higher level. To do that, we identify a number of policy-related questions that were asked in three countries – Germany Sweden and the Netherlands – at approximately the same time in the context of one VAA application and in national surveys. We find correspondence between the estimates of party-level policy preferences across both methods. However, we also find that coding decisions, such as including or excluding ‘Don’t knows’, can influence the agreement between the two estimates. Our findings are important for scholars who consider using VAA data for estimating public policy preferences. While our study does not provide a validation of VAA-derived estimates against an objective benchmark, the fact that two different methodological approaches converge on very similar estimates is highly suggestive of the potential of VAA data.