Resource Library
The BITSS Resource Library contains resources for learning, teaching, and practicing research transparency and reproducibility, including curricula, slide decks, books, guidelines, templates, software, and other tools. All resources are categorized by i) topic, ii) type, and iii) discipline. Filter results by applying criteria along these parameters or use the search bar to find what you’re looking for.
Know of a great resource that we haven’t included or have questions about the existing resources? Email us!
Disseminate
Collect & Analyze Data
IEEE DataPort Data Repositories+Metascience (Methods and Archival Science)Open PublishingReproducibility
PGRP Onboarding Materials for Collaborative Reproducible Workflows Data Management+EconomicsInterdisciplinaryPolitical ScienceReproducibilityVersion Control
Catalyst Thomas Brailey developed a set of training materials to help transition J-PAL’s Payments and Governance Research Program (PGRP) towards a version-controlled research pipeline by onboarding all research team members to GitHub, GitHub desktop, and R. These teaching materials can be applied to onboard other research/lab teams across a variety of contexts in social science research.
Coding style guides for collaborators (in R, Stata, and Python) Dynamic Documents and Coding Practices+InterdisciplinaryReproducibility
Developed by Sean Higgins (Northwestern University) and colleagues, this guide provides instructions for using R on research projects. Its purpose is to use with collaborators and research assistants to make code consistent, easier to read, transparent, and reproducible. See also the Python Guide and Stata Guide.
Researсh Ethics: History & Principles Data ManagementInterdisciplinary
Written by Lily Mayer, this infographic is a helpful introduction to the history and basic principles of research ethics.
Social Science Reproduction Platform Economics+Issues with transparency and reproducibilityMetascience (Methods and Archival Science)Other Social SciencesPolitical SciencePsychologyPublic HealthPublic PolicyReplicationsReproducibilitySociologyStatistics and Data Science
The Social Science Reproduction Platform crowdsources and catalogs attempts to assess and improve the computational reproducibility of social science research. Instructors can use the SSRP in applied social science courses at the graduate or undergraduate levels to teach fundamental concepts, methods, and reproducible research practices. Get started by creating a free account and browsing some of the completed reproductions! Instructors can start by reviewing the guide for instructors, which contains tips and resources for teaching and grading reproductions using the platform.
A template README for social science replication packages Data Management+EconomicsInterdisciplinaryOther Social SciencesPolitical SciencePsychologyPublic HealthPublic PolicyReproducibility
The template README follows best practices as defined by a number of data editors at social science journals. A full list of endorsers is listed in Endorsers. The most recent version is available at https://social-science-data-editors.github.io/template_README/. Specific releases can be found at https://github.com/social-science-data-editors/template_README/releases. The template README is available in a variety of formats, including HTML (best for reading), LaTeX, Word, PDF, and Markdown.
TIER Protocol 4.0 Data Management+InterdisciplinaryReproducibility
The TIER Protocol specifies the contents and organization of reproduction documentation for a project involving computations with statistical data.
Lab Manual for Jade Benjamin-Chung’s Lab Data Management+InterdisciplinaryPublic HealthReproducibility
ResearchBox Data Management+Interdisciplinary
ResearchBox offers an easy way to share and access scientific content, such as data, code, pre-registrations, and study materials. Uploaded files are organized into “Bingo Tables” that allow readers to easily find & access available files (e.g., researchbox.org/15). Among many features, ResearchBox provides:
- One-click downloads
- Instantaneous file-previews
- Codebooks for every dataset
- Integration with AsPredicted.org
Introduction to R (teaching materials) Data Visualization+Statistical Literacy
Fabio Votta (University of Stuttgart) shared teaching materials for an “Introduction to R” course, including slides, R basics script, and tidyverse script.
Open Research Calendar Data Management+Issues with transparency and reproducibilityOpen PublishingOpen ScienceReproducibilityStatistical Literacy
Open Research Calendar is an open-source community tool that collates information on worldwide events related to open science and research.
Development Research in Practice : The DIME Analytics Data Handbook Data Management+EconomicsEthicsImpact EvaluationInterdisciplinaryInternational DevelopmentPre-Analysis PlansPre-RegistrationStatistical Literacy
“Development Research in Practice” leads the reader through a complete empirical research project, providing links to continuously updated resources on the DIME Wiki as well as illustrative examples from the Demand for Safe Spaces study. The handbook is intended to train users of development data on how to handle data effectively, efficiently, and ethically. See an accompanying online course here.
An Introduction to Open Science Interdisciplinary+Open Science
This presentation by Felix Schönbrodt gives an overview of the motivation for open science and an introduction to the research tools and practices commonly associated with open science. The slides are can be re-used and distributed under the CC BY license.
Reproducible Data Science with Python Data Visualization+InterdisciplinaryReproducibilityStatistics and Data ScienceVersion Control
Written by Valentin Danchev, “Reproducible Data Science with Python” is a textbook that uses real-world social data sets related to the COVID-19 pandemic to provide an accessible introduction to open, reproducible, and ethical data analysis using hands-on Python coding, modern open-source computational tools, and data science techniques. Topics include open reproducible research workflows, data wrangling, exploratory data analysis, data visualization, pattern discovery (e.g., clustering), prediction & machine learning, causal inference, and network analysis.
Framework for Open and Reproducible Research Training (FORRT) Data Management+Dynamic Documents and Coding PracticesInterdisciplinaryIssues with transparency and reproducibilityPre-Analysis PlansStatistical LiteracyTransparent Reporting
FORRT is a pedagogical infrastructure designed to recognize and support the teaching and mentoring of open and reproducible science tenets in tandem with prototypical subject matters in higher education. FORRT also advocates for the opening of teaching and mentoring materials as a means to facilitate access, discovery, and learning to those who otherwise would be educationally disenfranchised.
Using the OSF (Open Science Framework) Open Science+Transparency
Find slides from a presentation by Johanna Cohoon and Caner Uguz titled “Using the OSF (Open Science Framework)”.
Data Sharing and Replication ReproducibilityTransparency
Find slides from a presentation by Garret Christensen titled “Data Sharing and Replication: Enabling Reproducible Research”.
Dataverse: Research Transparency through Data Sharing Data Repositories+ReproducibilityTransparency
Find slides from a presentation by Mercè Crosas titled “Dataverse: Research Transparency through Data Sharing”.
Comparing and Consolidating Empirical Findings Data Repositories
Find slides from a presentation by Solomon Hsiang titled “Comparing and Consolidating Empirical Findings”.
MCC, Microdata Protection, and Dissemination
Find slides from a presentation by Jennifer Sturdy titled “MCC, Microdata Protection, and Dissemination”.
Open Data and Protection of Research Subjects Social ScienceTransparency
Find slides from a presentation by Mercè Crosas titled “Open Data and Protection of Research Subjects”.
Building a Better Psychological Science Psychology
Find slides from a presentation by Eric Eich titled “Building a Better Psychological Science”.
Reporting Standards for Social Science Experiments Social ScienceTransparent Reporting
Find slides from a presentation by Kevin Esterling titled “Reporting Standards for Social Science Experiments”.
Ethics in Experimental Research Humanities+Psychology
Find slides from a presentation by Scott Desposato titled “Ethics in Experimental Research”.
What Scholars and Citizens Think of Experimental Ethics Ethics+InterdisciplinaryOther Social Sciences
Find slides from a presentation by Scott Desposato titled “What Scholars and Citizens Think of Experimental Ethics: Results of a Survey Experiment”.
Changing Incentives Toward Transparency Issues with transparency and reproducibilityTransparency
Find slides from a presentation by Brian Nosek titled “Changing Incentives Toward Transparency”.
Framing Transparency in Research: Issues and Opportunities Issues with transparency and reproducibility+Transparency
Find slides from a presentation by Victoria Stodden titled “Framing Transparency in Research: Issues and Opportunities”.
Promoting Transparency in the Social Sciences Issues with transparency and reproducibility+Social ScienceTransparency
Find slides from a presentation by Edward Miguel titled “Promoting Transparency in Social Science Research”.
BITSS Overview and Introduction to 2015 Annual Meeting
Find slides from a presentation by Edward Miguel titled “BITSS Overview and Introduction to 2015 Annual Meeting”.
Analysis Plans in Economics EconomicsPre-Analysis Plans
Find slides from a presentation by Benjamin Olken titled “Analysis Plans in Economics”.
Pre-Analysis Plans – Applications in Economics EconomicsPre-Analysis Plans
Find slides from a presentation by Katherine Casey titled “Pre-analysis Plans (PAPs): Applications in Economics”.
Data Adaptive Pre-Specification Statistics and Data Science
Find slides from a presentation by Maya Petersen titled “Data Adaptive Pre-Specification for Experimental and Observational Data”.
Perspectives from Biomedical Research Health Sciences+MedicinePre-Analysis PlansPre-RegistrationTransparent Reporting
Find slides from a presentation by Maya Petersen titled “Pre-Registration, Pre-analysis, and Transparent Reporting: Perspectives from biomedical research”.
False-Positives, p-Hacking, Power, and Evidential Value Statistics and Data Science
Find slides from a presentation by Leif Nelson titled “False-Positives, p-Hacking, Power, and Evidential Value”.
Publication Bias in the Social Sciences Open PublishingSocial Science
Find slides from a presentation by Neil Malhotra titled “Publication Bias in the Social Sciences”.
S-values: Conventional measures of the sturdiness of the signs regression coefficients Statistics and Data Science
Find slides from a presentation by Ed Leamer titled “S-values: Conventional measures of the sturdiness of the signs regression coefficients”.
Protocols That Work MethodologySocial Science
Find slides from a presentation by Nick Adams titled “Protocols That Work”.
Reproducible and Collaborative Statistical Data Science Pre-Analysis Plans
Find slides from a presentation by Philip Stark titled “Reproducible and Collaborative Statistical Data Science”.
Registration and Version Control with OSF & GitHub RegistriesVersion Control
Find slides from a presentation by Garret Christensen titled “Registration and Version Control with OSF & GitHub”.
Software and Workflow for Reproducible Research Reproducibility
Find slides from a presentation by Garret Christensen titled “ Software and Workflow for Reproducible Research”.
Tools and Resources for Data Curation Data Repositories
Find slides from a presentation by Stephen Abrams titled “Tools and Resources for Data Curation”.
How to write a Pre-Analysis Plan Pre-Analysis Plans
Find slides from a presentation by Dalson Figueiredo and Lucas Silva titled “How to write a Pre-Analysis Plan”.
Pre-Analysis Plans (French) Pre-Analysis Plans
Find slides from a presentation by Zachary Tsala Dimbuene titled “Pre-Analysis Plans (French)”.
Research Transparency Overview (French) Issues with transparency and reproducibility
Find slides from a presentation by Zachary Tsala Dimbuene titled “Research Transparency Overview (French)”.
Investigation of Data-Sharing Attitudes in the Context of a Meta-Analysis Metascience (Methods and Archival Science)Statistics and Data Science
Find slides from a presentation by Joshua Polanin titled “Investigation of Data-Sharing Attitudes in the Context of a Meta-Analysis”.
Implementing an RTR Strategy Issues with transparency and reproducibility
Find slides from a presentation by Arnaud Vaganay titled “Implementing an RTR Strategy”.
Drafting RTR Guidelines Issues with transparency and reproducibility
Find slides from a presentation by Arnaud Vaganay titled “Drafting RTR Guidelines”.
Research Transparency and Reproducibility (RTR) Issues with transparency and reproducibility
Find slides from a presentation by Arnaud Vaganay titled “Research Transparency and Reproducibility (RTR)”.
Open Scholarship for the Social Sciences Open PublishingSocial Science
Find slides from a presentation by Philip N. Cohen titled “Open Scholarship for the Social Sciences”.
Gates Open Research Interdisciplinary
Find slides from a presentation by the Center for Effective Global Action (CEGA) titled “Gates Open Research”.
The Strength of Evidence from Statistical Significance and P-values Statistics and Data Science
Find slides from a presentation by Dan Benjamin titled “The Strength of Evidence from Statistical Significance and P-values”.
Pre-Analysis Plans in Behavioral and Experimental Economics EconomicsPre-Analysis Plans
Find slides from a presentation by Johannes Haushoffer titled “Pre-Analysis Plans in Behavioral and Experimental Economics”.
Open Science Success Stories Data Management+Issues with transparency and reproducibility
The Open Research Funders Group curates the Open Science Success Stories, a database of examples of how openness has benefited researchers and broader society.
Data Citations module Data Management+InterdisciplinaryTransparent Reporting
Created by the Federal Reserve Bank of St. Louis, this module introduces students to the key elements of data citations. See also related modules for Data Literacy.
Handbook on Using Administrative Data for Research and Evidence-Based Policy Data Management+EconomicsInterdisciplinaryInternational DevelopmentReproducibility
Co-edited by Shawn Cole, Iqbal Dhaliwal, Anja Sautmann, and Lars Vilhuber and published by J-PAL’s Innovations in Data and Experiments for Action Initiative (IDEA), this handbook includes case studies of large-scale randomized evaluations using private and national government administrative data, and technical guidance to support partnerships with governments, nonprofits, or firms to access data and pursue cutting-edge, policy-relevant projects.
Survey of Registered Reports Editors Interdisciplinary+Results-Blind Review & Registered Reports
Between December 15, 2017 and January 31, 2018, BITSS surveyed the editors of 76 academic journals which at the time, accepted submissions in the Registered Report (RR) format. Find summary statistics of the results in this document.
CRediT (Contributor Roles Taxonomy) InterdisciplinaryTransparent Reporting
CRediT (Contributor Roles Taxonomy) is high-level taxonomy, including 14 roles, that can be used to represent the roles typically played by contributors to scientific scholarly output. The roles describe each contributor’s specific contribution to the scholarly output.
Comparison of multiple hypothesis testing commands in Stata Economics+Statistics and Data Science
In this post on the Development Impact blog, David McKenzie (World Bank) compares various Stata packages used for multiple hypothesis testing adjustments and discusses settings where each package is best applied.
Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) Educational Expansion Epidemiology+Statistical LiteracyTransparent Reporting
Created by Catalyst Melissa Sharp, this is an open-source repository for epidemiological research methods and reporting skills for observational studies, structured based on the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) Statement. Use it to discover new methods and reporting guidelines and contribute through the GitHub repository (https://github.com/sharpmel/STROBECourse/).
Pre-Analysis Plans for Observational Research Economics+Pre-Analysis Plans
In her presentation at RT2 DC in 2019, Fiona Burlig (University of Chicago) provides advice on how one can credibly pre-register an observational research project. Also see Burlig’s 2018 paper that describes three scenarios for pre-registration of observational work, including i) cases where researchers collect their own data; ii) prospective studies; and iii) research using restricted-access data.