The generalizability crisis by Dr Tal Yarkoni

About the speaker

Tal Yarkoni a Research Associate Professor in the Department of Psychology at the University of Texas at Austin. His academic work focuses on developing new tools and methods for the analysis of psychology and neuroimaging data. Tal mostly builds open-source software tools and writes methods-y papers about reproducibility, data standards, and best practices. He also has a background in substantive areas of psychology, including personality, executive control, and psycholinguistics.

About the talk

Most theories and hypotheses in psychology are verbal in nature, yet their evaluation overwhelmingly relies on inferential statistical procedures. The validity of the move from qualitative to quantitative analysis depends on the verbal and statistical expressions of a hypothesis being closely aligned—that is, that the two must refer to roughly the same set of hypothetical observations. Here I argue that most inferential statistical tests in psychology fail to meet this basic condition. I demonstrate how foundational assumptions of the “random effects” model used pervasively in psychology impose far stronger constraints on the generalizability of results than most researchers appreciate. Ignoring these constraints dramatically inflates false positive rates and routinely leads researchers to draw sweeping verbal generalizations that lack any meaningful connection to the statistical quantities they are putatively based on. I argue that the routine failure to consider the generalizability of one’s conclusions from a statistical perspective lies at the root of many of psychology’s ongoing problems (e.g., the replication crisis), and conclude with a discussion of several potential avenues for improvement.