Science Establishes New Statistics Review Board

The journal Science is adding an additional step of statistical checks to its peer-review process in an effort to strengthen confidence in published study findings.

From the July 4th edition of Science:

[…] Science has established, effective 1 July 2014, a Statistical Board of Reviewing Editors (SBoRE), consisting of experts in various aspects of statistics and data analysis, to provide better oversight of the interpretation of observational data. Members of the SBoRE will receive manuscripts that have been identified […] as needing additional scrutiny of the data analysis or statistical treatment. The SBoRE member assesses what the issue is that requires screening and suggests experts from the statistics community to provide it.

So why is Science taking this additional step? Readers must have confidence in the conclusions published in our journal. We want to continue to take reasonable measures to verify the accuracy of those results. We believe that establishing the SBoRE will help avoid honest mistakes and raise the standards for data analysis, particularly when sophisticated approaches are needed.

[…] I have been amazed at how many scientists have never considered that their data might be presented with bias. There are fundamental truths that may be missed when bias is unintentionally overlooked, or worse yet, when data are “massaged.” Especially as we enter an era of “big data,” we should raise the bar ever higher in scrutinizing the analyses that take us from observations to understanding.

Science is not the first academic journal to take new measures in the wake of growing concerns about data analysis mistakes and irreproducibility.

Political Science Research and Methods, for instance, now employs a data analyst who replicates the results of every empirical papers before final acceptance. In biomedicine, journals such as Annals of Internal Medicine, the Journal of the American Medical Association, and The Lancet are also known for paying strong attention to statistical review.

Columbia’s Chris Blattman said that researchers dealing with a lot of numbers are now expected to become better statisticians if they want to develop a substantive expertise in their field. This should start as early as undergraduate education, according to Blattman, where social science programs ought to include more courses on statistics and causal inference.