Expert- And LLM-Generated Data Inputs for the Prediction of Public Policies

Policy predictions with dynamic negotiation models, such as the Predictioneer’s Game and the DECIDE model, have traditionally relied on human generated data. These models explicitly and dynamically model complex negotiations among medium-sized to large bodies of actors, such as communal, national, EU, or global negotiations and have strong performance in prior applications On many policy issues (Bueno de Mesquita 2009, 2011; Wayman et al. 2014; Sprinz et al. 2016; Stokman and Van Oosten 1994; Thomson et al. 2006; Dijkstra et al. 2008), however, there are relatively few experts with precise knowledge of actor-specific data inputs for 20+ actors (incl. potential influence, position, salience on the issue, flexibility; i.e., 80+ data inputs per policy issue) – which limits replication of data. Expert-generated data are costly, time-intensive, and often need considerable lead time. Large Language Models (LLM) might mitigate these constraints. Initial development of such structured extraction pipelines is non-trivial; once established, they are potentially highly efficient and allow for near-time predictions – provided that safeguards for uncertainty, bias, and consistency are implemented. To the best of our knowledge, we are the first to undertake a systematic comparison of both methods of data in the context of policy prediction models. We undertook parallel data input generation for the Predictioneer’s Game on the future of a German federal government payment system for forest ecosystem services for 2027+ - which builds on existing funding systems since 2022 (Sprinz et al. 2024; Sprinz and Krott 2026). These payments are set to expire by calendar year 2026. This raises three questions: (1) Will the scope or breadth of the payment system change (incl. expiration)? (2) Will the stringency of requirements for payments be changed? And (3) Will the payments for actions be replaced by payments for results? All three issues are politically relevant, and formal negotiations have not commenced when we collected the data in the fall of 2025. In our paper, we will systematically compare human expert data generation with LLM-generated data inputs. For the latter, we will particularly attend to the identification of the relevant body of internet-accessible documents (scraper: access, extraction, and cleaning) and the prompt strategy for the various variables (incl. uncertainty awareness, bias steering, joint extraction of variables, and self-consistency prompting), and discuss the pros and cons of alternative choices. As the Predictioneer’s Game is a deterministic forecasting model, only differences in data inputs can account for differences in outcomes. Subjecting our human expert input data and LLM-generated data inputs to the Predictioneer’s Game results in two of the issues leading substantively to the same predicted policy forecasts (very narrow differences), yet the predictions differ by a quarter (10 points on a 0-100 scale) on the third issue. In the paper, we will extend on our current structured comparison of data inputs, elaborate on level effects between the data generating methods, variable-specific differences, as well as propose strategies for the improved use of LLM data generation in order to reduce resource costs and improve the reliability of near-time policy predictions.