Science Magazine Embarks on Reproducibility Quest

Marcia McNutt, Editor-in-Chief of Science, in the January 17 editorial:

Science advances on a foundation of trusted discoveries. Reproducing an experiment is one important approach that scientists use to gain confidence in their conclusions. Recently, the scientific community was shaken by reports that a troubling proportion of peer-reviewed preclinical studies are not reproducible. Because confidence in results is of paramount importance to the broad scientific community, we are announcing new initiatives to increase confidence in the studies published in Science. For preclinical studies (one of the targets of recent concern), we will be adopting recommendations of the U.S. National Institute of Neurological Disorders and Stroke (NINDS) for increasing transparency. Authors will indicate whether there was a pre-experimental plan for data handling (such as how to deal with outliers), whether they conducted a sample size estimation to ensure a sufficient signal-to-noise ratio, whether samples were treated randomly, and whether the experimenter was blind to the conduct of the experiment. These criteria will be included in our author guidelines […] Unfortunately, there are no equivalents to the NINDS recommendations for other disciplines that provide a basis for requiring transparency across all fields. For the next 6 months, we will be asking reviewers and editors to identify papers submitted to Science that demonstrate excellence in transparency and instill confidence in the results. This will inform the next steps in implementing reproducibility guidelines.

Read more…