Evaluating Replicability: Considerations for Analyses and Implications for Design

Abstract

As high-profile empirical research has questioned the replicability of scientific findings, it has become clear that there is no standard approach to designing and analyzing studies to evaluate replication. Ambiguity regarding key estimands for ‘replication’ and the purpose of replication research has shaped statistical treatments of the topic and sparked debate in several fields. This talk sheds light on this ambiguity by identifying different possible statistical definitions of ‘replication’ that could be studied. It then highlights relevant analysis methods and derives their statistical properties. Finally, it connects these properties to key implications for the design of primary studies, as well as subsequent replication attempts.

Jacob M. Schauer

Assistant Professor

My research interests involve statistical methods for the social and health sciences.