Assessing Sample Imbalances in Event-Study Designs: Observable and Unobservable Sources of Bias

Political events are well known to shape citizens’ attitudes, emotions, and political behavior. At the same time, survey methodology shows that external shocks can influence who responds to surveys and under which conditions. This paper bridges these literatures by arguing that political events may affect not only political outcomes, but also survey participation itself. When event-induced shifts in participation alter the composition of survey samples in ways that are correlated with outcomes of interest, estimates of an event’s causal effect may be biased. Such compositional bias may inflate, attenuate, or even mask the true substantive effect of the event. This challenge is particularly acute when compositional changes are driven by unobserved or unobservable characteristics that cannot be directly diagnosed or fully adjusted for using established analytical techniques. The paper focuses on the Unexpected Events during Survey Fieldwork (UESD) design, which is widely used to estimate causal effects of events by comparing respondents interviewed immediately before and after an event. I develop a framework to disentangle genuine causal effects from compositional bias arising from event-driven changes in survey participation. The framework proceeds in three steps. First, it formalizes how event-induced changes in responsiveness can generate bias even when fieldwork timing is as-if random. Second, it outlines strategies to detect and adjust for observable imbalances between pre- and post-event samples. Third, it extends sensitivity analyses to assess how strong unobserved confounding in survey participation would need to be to overturn substantive conclusions, thereby quantifying the potential for bias in estimates of event-effects. I illustrate the framework using the rally-around-the-flag effect, i.e. increased government approval in the aftermath of a severe shock, following the 2015 Charlie Hebdo attacks in Paris. I then apply the approach in a systematic replication of 14 published UESD studies examining terrorist events and rally-type outcomes such as government approval and political trust. Across cases, I show that accounting for both observable and unobservable sources of compositional bias can meaningfully affect estimated event-effects and, in some instances, challenge their robustness. By explicitly theorizing and diagnosing how political events can reshape survey samples, this paper strengthens causal inference in event-study designs and enhances the credibility of public opinion research conducted in dynamic and crisis-driven political contexts.