by Abel Brodeur
Introduction from BITSS: Today on the BITSS blog, Catalyst Abel Brodeur presents the Institute for Replication (I4R) (Twitter: @I4Replication), a new initiative for improving the credibility of published research. Read on to learn about I4R’s goals, its connections to the Social Science Reproduction Platform, and how to get involved.
Replication is key to credibility and confidence in research findings. Through their attempt to falsify past evidence, replications contribute in essential ways to the production of scientific knowledge. They allow us to assess which findings are robust, making science a self-correcting system, with major downstream effects on policy-making. Despite these benefits, reproducibility and replicability rates are surprisingly low, and direct replications (which repeat studies with the same methodology but different data) are rarely published. Addressing these challenges requires innovative approaches in how we conduct, reward, and communicate the outcomes of reproductions and replications.
That is why we are excited to announce the official launch of the Institute for Replication (I4R), a coordinated effort to improve the credibility of science by systematically reproducing and replicating research findings in leading academic journals. Our team of collaborators supports researchers and aims to improve the credibility of science by:
- Reproducing, conducting sensitivity analysis of, and replicating studies published in leading journals.
- Establishing an open-access website to serve as a central repository containing the replications, responses by the original authors, and documentation.
- Developing and providing access to educational materials on replication and open science.
- Preparing standardized file structure, code ,and documentation aimed at facilitating reproducibility and replicability by the broader community.
How I4R works
Our primary goal is to promote and generate replications. We use the term “replication” as an umbrella term for different approaches. Replications may be achieved using the same or different data and procedures, and a variety of definitions are being used.
We are actively looking for replicators and have an ongoing list of studies we’re looking to be replicated. We have already checked that the codes and data provided by the selected studies are sufficient to reproduce their results (i.e., run the code). This allows replicators to focus on conducting sensitivity analysis and/or replicating the results using the raw data.
Our collaborators provide detailed instructions on how to conduct a replication in the Guide for Accelerating Computational Reproducibility in the Social Sciences. We also developed a template for writing replications which is available here. This template provides examples of robustness checks and how to report the replication results.
Replicating papers could be useful for instructors looking for class projects, and also for junior and senior researchers who want to improve their knowledge of some leading research papers by going through them in detail. Replicators may decide to remain anonymous. The decision to remain anonymous can be made at any point during the process; initially, once completed, or once the original author(s) provided an answer.
We will provide assistance for helping replicators publish their work. Replicators will also be invited to co-author a large meta-analysis paper that will combine the work of all replicators and answer questions such as which type of studies replicate and what characterizes results that replicate. For more on publishing replications, keep reading!
We need your help
I4R is open to all researchers interested in advancing the reproducibility and replicability of research. We need your help reproducing and replicating as many studies as possible! Please contact us if you are interested in getting involved. We are also actively looking for researchers with large networks to serve on the editorial board, especially in the field of macroeconomics for economics and international relations for political science.
Beyond helping out with replication efforts, you can help our community by bringing replication to your classroom, e.g. using the Social Science Reproduction Platform (SSRP). A list of educational resources is available here.
Want to get involved? Have thoughts? Want to replicate a study, reach out to us! If you’re an instructor, use the SSRP in your class!
Where to publish replications
Incentives for replications are currently limited, with a small number of replications published in top journals. Moreover, reproducing or replicating others’ work can lead to disagreements with the original author(s) whose work is re-analyzed. One of I4R’s main objectives is to address these challenges and help researchers conduct and disseminate reproductions and replications.
As a first step to better understand publication possibilities for replicators, our collaborators (Jörg Peters and Nathan Fiala) and the Chair, Abel Brodeur, surveyed over 50 journal editors in economic, finance, and political science about their interest in replicating and reporting their full answers on our website. Our survey consists of two questions:
- Do you publish comments in the [journal name]? By comment, we mean a paper that discusses and potentially challenges the empirical results from another paper, for example, based on a reanalysis or additional robustness checks.
- If yes, do you only publish comments on original papers that have previously been published in the [journal name], or do you also publish comments on original papers that have been published elsewhere?
We find that the majority of editors answer Yes to the first question. We also find that many editors answer Yes to the second question, but that the replication would go through the regular review process and would need to make a contribution of general interest.
We also highlight special issues/symposiums dedicated to replications and journals which strictly publish comments. Please contact us if you want to advertise other replication efforts or special issues related to open science and replications.
We will continue developing new and exciting features based on input from the community. Do not hesitate to reach out to us!
Abel Brodeur is an Associate Professor in the Department of Economics at the University of Ottawa. Dr. Brodeur received a 2016 Leamer-Rosenthal Prize for Open Social Science for his work on the ground-breaking paper “Star Wars: The Empirics Strike Back” published in the American Economic Journal: Applied Economics.