By Fernando Hoces de la Guardia (BITSS) and Sebastián Martínez (Inter-American Development Bank)

This post is cross-posted on the IDB Impact blog and the CEGA blog. You can read it in Spanish here.

Introduction from BITSS: This post highlights the results of a successful partnership between BITSS and the Inter-American Development Bank (IDB). Following a BITSS workshop in 2018, discussions about how the IDB could provide more sustained support to its staff led to a collaborative effort to adapt best practices for transparency, reproducibility, and ethics into formal standards for IDB research. The resulting technical note, discussed below, reviews higher standards for rigor and ethics in the context of actual practice and oversight.

While training, funding of meta-science, recognition of exceptional individuals, and community-building continues to be the bedrock of BITSS’s mission, we recognize that research is conducted within a larger ecosystem in which funding, publishing, and career incentives placed on individuals don’t always align with the interests of the wider scientific community. Such recognition is also rising among research organizations, journals, academic associations, and funders who are starting to seek training and technical support to address misaligned incentives. In response, we’ve begun partnering with institutions, balancing our bottom-up efforts with top-down systemic transformation to enable adoption of best practices at scale.

If your institution is interested in receiving similar support, contact BITSS Program Manager Katie Hoeberling (khoeberling@berkeley.edu).

---

The standards of what constitutes rigorous research are rising. In a response to what has been called a credibility crisis, entire scientific fields are calling for published research to more closely follow the scientific ideals most scientists learn early in their education. Following suit, the IDB, a leader in the generation of knowledge for Latin America, partnered with the Berkeley Initiative for Transparency and Social Sciences (BITSS) to survey the state of the art in best practices for Transparent, Reproducible, and Ethical research. These best practices are relevant for a variety of research activities including surveys, evaluations, economic analyses and other applied research initiatives.

The three core principles are defined as follows:

• Transparent research refers to the set of practices and tools used to disclose all methods and data behind an analysis.
• Reproducible research refers to the ability of the research community to access and recreate the final results of the analysis from raw data with minimal effort.
• Ethical research refers to practices following the principles of beneficence, respect for persons, and justice.

In addition to the personal satisfaction of practicing science in a more transparent, reproducible, and ethical way, incentives in the conduct and publishing of research are aligning more and more with these new standards. Below are three reasons in favor of incorporating these practices into your own research.

Reason #1:  Do you want to publish your next randomized controlled trial (RCT) in a top economics journal?

If so, you will need to register your experiment in an online registry, such as the socialscienceregistry.org. This short description of a study is now a requirement by the American Economic Association, and it may soon be required for all prospective empirical studies, on top of RCTs. After registration, a more detailed Pre-Analysis Plan (PAP) can help you clearly differentiate between which hypotheses you decide to posit and test before looking at data, and which you may explore after looking at data. In addition to reducing the risk of using questionable research practices, writing a detailed PAP may help you publish your paper in the Journal of Development Economics. The JDE is piloting a new kind of article – Registered Reports – providing in-principle acceptance for publication regardless of how eye-catching the final results are.

Reason #2: Do you want to avoid sleepless nights a few years from now when you try to reproduce tables from one of your papers?

Then don’t forget to set up a file management structure, use version control, and publish the code and data that will allow you (or anyone else) to quickly arrive at the same coefficient in your published paper. Or better yet, adopt 21st century technology and integrate your entire research project with code embedded directly into your paper with dynamic documents, available in R, Python, and Stata. In addition to helping your future self, you also will increase your chances of publishing in a top journal, as the AEA has also begun to focus much more on the issue of computational reproducibility.

Reason #3: Do you want to make sure your research is ethically conducted, protects participants, and minimizes the reputational risk for yourself and for your institution? Following best practices for ethical research can help ensure that your research espouses the principles of respect for persons, beneficence, and justice, but can also create tension with other practices for transparent and reproducible research – particularly for open data and code. Ethical research practices include undergoing training on protections for human subjects, an independent review of the research protocol and writing a data management plan to clearly define how your project’s data will be safely collected, managed, stored, and de-identified for dissemination.

These best practices, and more, are outlined in detail in the new IDB Technical Note on Transparent, Reproducible and Ethical Research. While there is a lot of information to unpack, the principles are organized in an easy-to-follow checklist of activities that researchers can implement at each phase of a research project, from design through implementation and final dissemination. So even if you are half way through or near completion of your project, there are likely simple steps that you can still implement. And as you discover other reasons for adopting best practices in transparent, reproducible and ethical research, share them with us in the comments section below.

---

Fernando Hoces de la Guardia is a Postdoctoral Fellow at BITSS, leading the development of a broad agenda for transparency and reproducibility for policy analysis.

Sebastian Martínez is an Economics Principal Specialist at the Office of Strategic Planning and Development Effectiveness at the Inter-American Development Bank.