Psychology Open Science Progress Report

Etienne LeBel–BITSS Project Scientist


My name is Etienne LeBel and I’m a new Project Scientist at BITSS. My background is in social psychology (areas: racial biases/stereotypes, self, & attitudes/persuasion), though most of my recent work involves meta-scientific contributions in the emerging open science and replication movement in psychology. My primary role at BITSS is to lead the design and implementation of a longitudinal study tracking social scientists’ evolving attitudes about transparency, reproducibility, and replicability practices and testing experimental interventions to increase prevalence rates of actual transparency behaviors. Hitting the ground running, I represented BITSS at two psychology conferences within my first week of work, attending the annual meeting for the Association for Psychological Science (APS; Chicago, IL, May 26-29, 2016) and the inaugural meeting of the Society for Improvement of Psychology Science (SIPS; Charlottesville, VA, @ COS, June 5-8, 2016). Below is a “Progress report and next steps” summary of key open science categories that emerged at these two conferences.

Open Data/Materials

  • Open data badges at Psychological Science lead to 10-fold increase in reported open data (i.e., 22.8% articles after badges vs. 2.5% before badges and 2.8% at control journals; see article & graph below; similar though less dramatic effect for open materials)
    open-data-badges-journal.pbio.1002456.g002
  • Next steps:
    • Open data policy designed to be universally inclusive from the start (Srivastava @ SIPS)
    • New ways to recognize/credit non-traditional diverse contributions (e.g., open data for difficult to collect samples) via citation counts for data, # of downloads, “usability likes” (Condon, SIPS breakout group)

Pre-registration

Reproducibility

  • Overall reproducibility rate of published results in psychology not yet estimated, however anecdotal evidence suggests fairly low rates of reproducibility (which accords with ~25-40% reproducibility rates found in Economics [see Fiedler @APS])
  • Next steps:
    • New reproducibility badge proposed @SIPS, which resonates well with Fiedler’s point that reproducibility checks represent alternative open science contributions for researchers lacking access to subject pools (e.g. reproducibility check UI prototype)

Replications

Peer Review

  • Automated verification of statistical reporting errors at Psychological Science via StatCheck (SIPS)!
  • Next steps:
    • Results blind reviewing and promoting ethical citation and editorial practices via Editor Ethics 2.0 initiative (Banks)
    • New tools & heuristics to better estimate evidentiary value when peer-reviewing (Simmons)

Meta-Theoretical

  • More focus on updating state of theory with respect to follow-up (replication) studies (Lilienfield; e.g., Strength model of self-control)
  • Increased focus on intra-individual designs (Shoda; see also this working paper) and phenomenon-based investigations (Morey slides & video)
  • Better training for designing studies that yield informative results independent of outcome (Vazire, APS)
  • Increasing efficiency of knowledge creation (Simons); replications very costly, perhaps we should adopt more forward-thinking approach & focus primarily on open & reproducible findings (see also Coyne)
  • Tradeoffs and increasing open science uptake by striving for smaller/more incremental improvements (e.g., sufficiently open science, SIPS)

Culture/Norms

  • Widening the open science tent by increasing diversity of opinions and correcting “outsider” misperceptions (new website/FAQ emerging from SIPS breakout group)
  • Increased focus on positive benefits & examples of open science practices rather than negative criticism & identifying QRPs (indeed, but fundamentally scientific spirit involves ruthless skepticism & falsification)

As you can see, tons of exciting developments happening in psychology, with growing evidence that open science initiatives are having major positive impacts. However, many challenges remain and lots of work is ahead of us, but at least there’s a lot of optimism in the air!

 

See also other write-ups about SIPS by Wyble and Danvers.

Leave a Reply