Creating Standards for Reproducible Research: Overview of COS Meeting

By Garret Christensen (BITSS)

Representatives from BITSS (CEGA Faculty Director Ted Miguel, CEGA Executive Director Temina Madon, and BITSS Assistant Project Scientist Garret Christensen–that’s me) spent Monday and Tuesday of this week at a very interesting workshop at the Center for Open Science aimed at creating standards for promoting reproducible research in the social-behavioral sciences. Perhaps the workshop could have used a catchier name or acronym for wider awareness, but we seemed to accomplish a great deal.  Representatives from across disciplines (economics, political science, psychology, sociology, medicine), from funders (NIH, NSF, Laura and John Arnold Foundation, Sloan Foundation), publishers (Science/AAAS, APA, Nature Publishing Group), editors (American Political Science Review, Psychological Science, Perspectives on Psychological Science, Science), data archivists (ICPSR), and researchers from over 40 leading institutions (UC Berkeley, MIT, University of Michigan, University of British Columbia, UVA, UPenn, Northwestern, among many others) came together to push forward on specific action items researchers and publishers can do to promote transparent and reproducible research.

The work was divided into five subcommittees:

1) Reporting standards in research design

2) Reporting standards in analysis

3) Replications

4) Pre-Registration/Registered Reports

5) Sharing data, code, and materials

Over the course of two days, each subcommittee presented twice, first with a longer statement, and second with brief action statements that journals could sign on to, with three varying levels of difficulty. Understanding that reproducibility will involve some costs to researchers, and there are those in opposition (including some in the room), we wanted there to be relatively simple items that practically everyone could agree on. These statements aren’t finalized yet, but we’ll share them as soon as they are.

Here are some things that particularly interested me:

Data Sharing

  • You want to share your data so that others can replicate your work, but how do you do that when an integral part of your analysis is the GPS coordinates of the respondents’ homes? This interested me since some work of mine from the WASH Benefits project clearly uses geolocations to calculate distances from respondents homes to their water source. In order to do a pure replication of my work, one would need access to personal identifying information, and I can’t give that to you without violating my IRB protocol. One proposed solution mentioned by JPAL’s Rachel Glennerster was that on future projects I could write an IRB protocol that allowed me to share the data with researchers at other institutions, provided that they themselves obtained approval from their home institution IRB.
  • Political science journals seem to be pretty far out in front on data sharing: see the DA-RT (Data Access and Research Transparency) statement at http://www.dartstatement.org, and follow them on Twitter at https://twitter.com/DARTsupporters.
  • It’s ideal to use a trusted repository (Dataverse, OpenICPSR) to post your data in lieu of your own academic website, since you may change universities or your personal website may disappear. Also the data repositories can take your proprietary format data (SAS, Stata, etc.) and make it openly accessible, without losing metadata such as variable labels and value labels, as you might if you just throw a .csv file up on your personal website.

Reporting Standards

Uri Simonsohn of the University of Pennsylvania urged that checkboxes are definitely not enough to ensure that researchers have met all the reporting requirements of any CONSORT-like reporting/disclosure standard that social science researchers hope to follow. It’s just human nature to check the box without even thinking about it. Instead, it’s better to require copying and pasting of the relevant section from the paper as proof that the requirement is met, then this checklist with copy and pasted text could be sent to reviewers along with the paper, and reviewers could more easily determine that everything necessary had been disclosed.

Replication

How do we encourage replication, and are replications worthy of publication in the same journal as the original paper? Are replications that contradict the original paper of more value than replications that bolster the original? There was some disagreement on this point. Some think that failures to replicate are more valuable, but if that’s the case, and successful replications don’t get any attention, how do we prevent the public from casting aspersions about entire bodies of literature based on a few failures to replicate? Take the Reinhart and Rogoff controversy as an example. Just because that one paper contained an error does not mean that all economics papers contain errors. Thomas Herndon replicated R&R’s work as an assignment in a graduate school course. I haven’t heard anything about the replications by all the other students in the class. Were they successful?

Pre-Registration?

We apparently need to work out some semantic issues in the transparency movement, because “registering” means different things to different people.

  • In my mind, “registering” is the act of declaring that you are running a randomized trial. Register your trial with clinicaltrials.gov or socialscienceregistry.org, for example. But some call this “pre-registering” which then confuses the act of registering a study with writing a Pre-Analysis Plan.
  • A Pre-Analysis Plan (PAP) is a detailed specification of the hypotheses you plan to test, and how you plan to test them, written before you start looking at your final data.
  • All this talk of registering can also be confused with “Registered Reports,” a term I hadn’t heard before this meeting. I’d always heard of this as “Results-blind reviewing.” Results-blind reviewing is the idea of submitting to a journal a study hypothesis and design before the experiment and analysis have taken place. Then the journal can give an in-principle acceptance (or rejection) based on the value of your hypothesis and the design of your study, instead of the significance of your results. If you follow through on your design and get a null result, your paper will still be published, because the truth might actually be zero. A lot of great work is being done on this by Chris Chambers and you should check out the Registered Reports page on the Open Science Framework.

Positive Recognition

There was a lot of discussion of the need for positive reinforcement of transparent research instead of finger-pointing at non-compliance. Prizes could be given by professional societies, journals, or other organizations to reward those whose research exemplifies the transparent, reproducible ideal.

Some of that is already being done in the form of badges. The Center for Open Science (https://osf.io/tvyxz/) has developed three badges for Open Data, Open Materials, and Pre-Registered. Journals have signed on to award these badges—if your paper is being published in Cortex or Psychological Science, for example, apply for the relevant badge, the editors and reviewers will see if the criteria fits, and if so, award the badge to the paper. (I’m an Eagle Scout, so obviously I like the merit badges idea.)

 

thumb_garret_photoAbout the author: Garret Christensen joined the BITSS team as a project scientist in September of this year. He earned his PhD in Economics from UC Berkeley in 2011, has worked on child health and education randomized trials in Kenya, and taught economics at Swarthmore College. He has also run more than forty ultra-marathons and walked across the entire United States four times.