The BITSS Resource Library contains resources for learning, teaching, and practicing research transparency and reproducibility, including curricula, slide decks, books, guidelines, templates, software, and other tools. All resources are categorized by i) topic, ii) type, and iii) discipline. Filter results by applying criteria along these parameters or use the search bar to find what you’re looking for.
Know of a great resource that we haven’t included or have questions about the existing resources? Email us!
Lab Manual for Jade Benjamin-Chung’s Lab Data Management and De-identificationInterdisciplinaryPublic HealthReproducibility
ResearchBox Data Management and De-identificationInterdisciplinary
ResearchBox offers an easy way to share and access scientific content, such as data, code, pre-registrations, and study materials. Uploaded files are organized into “Bingo Tables” that allow readers to easily find & access available files (e.g., researchbox.org/15). Among many features, ResearchBox provides:
- One-click downloads
- Instantaneous file-previews
- Codebooks for every dataset
- Integration with AsPredicted.org
Open Research Calendar Data Management and De-identificationIssues with transparency and reproducibilityOpen PublishingOpen ScienceReproducibilityStatistical Literacy
Open Research Calendar is an open-source community tool that collates information on worldwide events related to open science and research.
Development Research in Practice : The DIME Analytics Data Handbook Data Management and De-identificationEconomicsEthicsImpact EvaluationInterdisciplinaryInternational DevelopmentPre-Analysis PlansPre-RegistrationStatistical Literacy
“Development Research in Practice” leads the reader through a complete empirical research project, providing links to continuously updated resources on the DIME Wiki as well as illustrative examples from the Demand for Safe Spaces study. The handbook is intended to train users of development data on how to handle data effectively, efficiently, and ethically. See an accompanying online course here.
An Introduction to Open Science InterdisciplinaryOpen Science
This presentation by Felix Schönbrodt gives an overview of the motivation for open science and an introduction to the research tools and practices commonly associated with open science. The slides are can be re-used and distributed under the CC BY license.
Reproducible Data Science with Python Data VisualizationInterdisciplinaryReproducibilityStatistics and Data ScienceVersion Control
Written by Valentin Danchev, “Reproducible Data Science with Python” is a textbook that uses real-world social data sets related to the COVID-19 pandemic to provide an accessible introduction to open, reproducible, and ethical data analysis using hands-on Python coding, modern open-source computational tools, and data science techniques. Topics include open reproducible research workflows, data wrangling, exploratory data analysis, data visualization, pattern discovery (e.g., clustering), prediction & machine learning, causal inference, and network analysis.
Framework for Open and Reproducible Research Training (FORRT) Data Management and De-identificationDynamic Documents and Coding PracticesInterdisciplinaryIssues with transparency and reproducibilityPre-Analysis PlansStatistical LiteracyTransparent Reporting
FORRT is a pedagogical infrastructure designed to recognize and support the teaching and mentoring of open and reproducible science tenets in tandem with prototypical subject matters in higher education. FORRT also advocates for the opening of teaching and mentoring materials as a means to facilitate access, discovery, and learning to those who otherwise would be educationally disenfranchised.
Dataverse: Research Transparency through Data Sharing Data RepositoriesReproducibilityTransparency
Find slides from a presentation by Mercè Crosas titled “Dataverse: Research Transparency through Data Sharing”.
Reporting Standards for Social Science Experiments Social ScienceTransparent Reporting
Find slides from a presentation by Kevin Esterling titled “Reporting Standards for Social Science Experiments”.
What Scholars and Citizens Think of Experimental Ethics EthicsInterdisciplinaryOther Social Sciences
Find slides from a presentation by Scott Desposato titled “What Scholars and Citizens Think of Experimental Ethics: Results of a Survey Experiment”.
Framing Transparency in Research: Issues and Opportunities Issues with transparency and reproducibilityTransparency
Find slides from a presentation by Victoria Stodden titled “Framing Transparency in Research: Issues and Opportunities”.
Find slides from a presentation by Edward Miguel titled “BITSS Overview and Introduction to 2015 Annual Meeting”.
False-Positives, p-Hacking, Power, and Evidential Value Statistics and Data Science
Find slides from a presentation by Leif Nelson titled “False-Positives, p-Hacking, Power, and Evidential Value”.
S-values: Conventional measures of the sturdiness of the signs regression coefficients Statistics and Data Science
Find slides from a presentation by Ed Leamer titled “S-values: Conventional measures of the sturdiness of the signs regression coefficients”.
Reproducible and Collaborative Statistical Data Science Pre-Analysis Plans
Find slides from a presentation by Philip Stark titled “Reproducible and Collaborative Statistical Data Science”.
Registration and Version Control with OSF & GitHub RegistriesVersion Control
Find slides from a presentation by Garret Christensen titled “Registration and Version Control with OSF & GitHub”.
Investigation of Data-Sharing Attitudes in the Context of a Meta-Analysis Metascience (Methods and Archival Science)Statistics and Data Science
Find slides from a presentation by Joshua Polanin titled “Investigation of Data-Sharing Attitudes in the Context of a Meta-Analysis”.
The Strength of Evidence from Statistical Significance and P-values Statistics and Data Science
Find slides from a presentation by Dan Benjamin titled “The Strength of Evidence from Statistical Significance and P-values”.
Pre-Analysis Plans in Behavioral and Experimental Economics EconomicsPre-Analysis Plans
Find slides from a presentation by Johannes Haushoffer titled “Pre-Analysis Plans in Behavioral and Experimental Economics”.
Handbook on Using Administrative Data for Research and Evidence-Based Policy Data Management and De-identificationEconomicsInterdisciplinaryInternational DevelopmentReproducibility
Co-edited by Shawn Cole, Iqbal Dhaliwal, Anja Sautmann, and Lars Vilhuber and published by J-PAL’s Innovations in Data and Experiments for Action Initiative (IDEA), this handbook includes case studies of large-scale randomized evaluations using private and national government administrative data, and technical guidance to support partnerships with governments, nonprofits, or firms to access data and pursue cutting-edge, policy-relevant projects.
Survey of Registered Reports Editors InterdisciplinaryResults-Blind Review & Registered Reports
Between December 15, 2017 and January 31, 2018, BITSS surveyed the editors of 76 academic journals which at the time, accepted submissions in the Registered Report (RR) format. Find summary statistics of the results in this document.
CRediT (Contributor Roles Taxonomy) InterdisciplinaryTransparent Reporting
CRediT (Contributor Roles Taxonomy) is high-level taxonomy, including 14 roles, that can be used to represent the roles typically played by contributors to scientific scholarly output. The roles describe each contributor’s specific contribution to the scholarly output.
Comparison of multiple hypothesis testing commands in Stata EconomicsStatistics and Data Science
In this post on the Development Impact blog, David McKenzie (World Bank) compares various Stata packages used for multiple hypothesis testing adjustments and discusses settings where each package is best applied.
Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) Educational Expansion EpidemiologyStatistical LiteracyTransparent Reporting
Created by Catalyst Melissa Sharp, this is an open-source repository for epidemiological research methods and reporting skills for observational studies, structured based on the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) Statement. Use it to discover new methods and reporting guidelines and contribute through the GitHub repository (https://github.com/sharpmel/STROBECourse/).
Pre-Analysis Plans for Observational Research EconomicsPre-Analysis Plans
In her presentation at RT2 DC in 2019, Fiona Burlig (University of Chicago) provides advice on how one can credibly pre-register an observational research project. Also see Burlig’s 2018 paper that describes three scenarios for pre-registration of observational work, including i) cases where researchers collect their own data; ii) prospective studies; and iii) research using restricted-access data.
Data for Development Impact (Resource Guide) Data Management and De-identificationEconomicsOther Social SciencesStatistics and Data Science
“Data for Development Impact: The DIME Analytics Resource Guide” is intended to serve as an introduction to the primary tasks required in development research, from experimental design to data collection to data analysis to publication. It serves as a companion to the DIME Wiki and is produced by DIME Analytics.
Open Science Module for Behavioral Science graduate course EconomicsPsychology
Instructors Kelly Zhang (MIT GOV/LAB) and Chaning Jang (Busara) integrated a module on research transparency and the use of pre-analysis plans as part of the Behavioral Science in the Field course designed for graduate students who use behavioral science games as part of their research.
J-PAL Guide to De-Identifying Data Data Management and De-identificationInternational Development
Developed by J-PAL’s Sarah Kooper, Anja Sautmann, and James Turrito, this guide includes:
- An overview of personally identifiable information (PII) and the responsibility of data users not to use data to try to identify human subjects
- Recommendations for handling direct identifiers (such as full name, social security number, or phone number), as well as indirect identifiers (such as month/year of birth, nationality, or gender)
- Guidance on de-identification steps to take throughout the research process, such as encrypting all data containing identifying information as soon as possible
- A list of common identifiers, including those labeled by the United States’ Health Insurance Portability and Accountability Act (HIPAA) guidelines as direct identifiers
- And more.
See also the accompanying Guide to Publishing Research Data.
J-PAL Guide to Publishing Research Data Data Management and De-identificationInternational DevelopmentPublic Policy
Developed by J-PAL’s Sarah Kooper, Anja Sautmann, and James Turrito, this guide includes:
- A list of considerations to make before publishing data, such as what information was provided to study participants and the IRB, the sensitivity of the data collected, and legal requirements
- Sample consent form language that will allow future publication of de-identified data
- A checklist for preparing data for publication
- And more.
See also the accompanying Guide to De-identifying Data.
Data Sharing Checklist for NGOs and Practitioners Data Management and De-identificationInterdisciplinary
This checklist developed by Teamscope can help NGOs and Practitioners understand the common pitfalls in open data, and how open data impacts every step of a project’s pipeline, from proposal writing to dissemination.
Videos: Research Transparency and Reproducibility Training (RT2) – Washington, D.C. Data Management and De-identificationInterdisciplinaryIssues with transparency and reproducibilityMeta-AnalysesPower analysisPre-Analysis PlansPreprintsRegistriesReplicationsResults-Blind Review & Registered ReportsStatistical LiteracyTransparent ReportingVersion Control
BITSS hosted a Research Transparency and Reproducibility Training (RT2) in Washington DC, September 11-13, 2019. This was the eighth training event of this kind organized by BITSS since 2014.
RT2 provides participants with an overview of tools and best practices for transparent and reproducible social science research. Click here to videos of presentations given during the training. Find slide decks and other useful materials on this OSF project page (https://osf.io/3mxrw/).
Preregistration of secondary data analysis: A template and tutorial InterdisciplinaryRegistries
Van den Akker and colleagues present a template specifically designed for the preregistration of secondary data analyses and provide comments and a practical example.