The Devil is in the Details: Using Machine-Learning to Scrutinize "State-of-the Art" Language Models’ Responses to Public Inquiries across 3 Continents

This research project seeks to investigate bias, quality, and coherence of the so called "state-of-the art" Artificial Intelligence (AI) driven Language Models. In this iteration of the project, the researchers will evaluate OpenAI’s ChatGPT model and its responses to a diverse group of respondents spanning North America, Europe, and Africa. In an era where AI-driven language models like ChatGPT are increasingly influencing decision-making and shaping opinions, it is crucial to assess their advice and guidance, particularly on issues of public interest. The study will employ a non-representative large sample of university students, drawn from various educational institutions across the selected continents. While no representative sampling techniques will be employed, the diverse geographical and cultural backgrounds of the participants from North American, European and African Universities will provide a rich and varied dataset that will be leveraged in a rigorous comparative analysis. The research will focus on ChatGPT's responses to critical topics in public administration including paying taxes, respecting government orders/directives, indigenous rights, indigenous self-determination, whistleblowing, participating in the electoral process, racism, and pursuing careers in public management. Using advanced machine-learning techniques, including classification, natural language processing (NLP), and sentiment analysis, the study will assess the coherence and topical orientation of ChatGPT's advice across the three continents on the selected topics. Comparative analysis will help identify regional variations in the model’s responses. Furthermore, the research will integrate theoretical insights from the field of public administration to provide a comprehensive evaluation of ChatGPT's guidance. By scrutinizing how the language model navigates the nuances and complexities of these topics, the study aims to shed light on the potential societal impact of AI-driven chatbots on the decision-making processes of the respondents. With such a diverse group of respondents, this research will contribute to our understanding of the implications of AI-driven language models in shaping the perspectives and individual choices across different continents, with a focus on these critical topics of public administration. The findings will not only inform the development of AI-driven conversational agents but also offer valuable insights for policymakers, educators, and the society.