Study results that cannot be replicated—i.e., confirmed as accurate—have made science vulnerable to criticism. But is scientific misconduct solely responsible for the replication crisis? Researchers from the joint project “Conceptual Replications” at the Leibniz Institute for Educational Trajectories (LIfBi) have identified other factors and now offer some sort of blueprint for explaining differences in research findings.
It is a problem that many findings in psychology, medicine, and the social sciences cannot be confirmed in replication studies. Key findings of these disciplines are being called into question. Scientists are suspected of deliberately influencing results. However, there are other reasons for effect heterogeneity.
In the joint project “Conceptual Replications: Guidelines for Implementation and Factors Influencing Replicability in Different Psychological Disciplines,” researchers from LIfBi, Freie Universität Berlin, Universität zu Köln, and Ludwig-Maximilians-Universität München took a closer look at the design and analysis of replication studies. They concluded that many replication studies lack information on certain study characteristics. For example, little is reported about recruitment methods, participant characteristics, or the environmental conditions under which the studies were conducted. “The focus in replication studies traditionally seems to be primarily on repeating material and analysis methods, while other important characteristics are neglected,” says Dr. Marie-Ann Sengewald, project manager at LIfBi.
In controlled replication studies conducted by the researchers themselves, the composition of the sample proved to be particularly relevant for different research findings. Based on their findings, the researchers developed recommendations regarding how replication studies can be implemented more systematically. “We want to provide some sort of blueprint for identifying causal reasons for different study results. To this end, we have developed controlled designs, clearly defined variations, and analysis methods to rule out confounding factors in replication studies,” explains Marie-Ann Sengewald. Information on the preliminary project results is available online on the project page. Conceptual Replications.
The researchers now want to extend their findings to disciplines where non-experimental studies are more commonly used. A follow-up application is planned. The Conceptual Replications project was funded by the German Research Foundation as part of the META-REP priority program.
Detailed insights into background information and the researchers' approach can be found in the interview with Dr. Marie-Ann Sengewald.
To the interview as PDF