Confirmation Bias in Probabilistic Process Tracing

Since its emergence in the late 1970s, process tracing has evolved substantially, adapting to diverse data forms and contemporary research practices. These methodological developments, however, come with new challenges: older and newer versions of the methodology coexist, blurring epistemological boundaries. While such evolution is desirable, it also increases the risk that newer probabilistic testing tools are incorrectly applied within older, non-probabilistic research designs. Accordingly, this article examines the misapplication of probabilistic process tracing, focusing on a specific form of confirmation bias. This bias arises when a researcher tests a hypothesis with the same data used to derive it, thereby retrofitting the testing design ex post. This can inflate posterior probabilities through double counting and, consequently, lead to overconfidence in the inferred mechanism. Moreover, the identified bias may diminish the capacity to distinguish between competing hypotheses. It is argued that this bias is rooted in internal tensions within the method itself, as it imposes the linearity of causal reasoning onto the inherently simultaneous nature of qualitative research practice. Multiple strategies to address this problem are proposed, including a stronger emphasis on assessing priors and establishing a clearer division between different types of evidence.