To Get More Out of Science, Show the Rejected Research


In a recent opinion piece on the New York Times news portal the Upshot, Brendan Nyhan, an assistant professor of government at Dartmouth College, comments on a host of transparency related issues.

Closely echoing the mission of BITSS, Nyhan identifies the potential of research transparency to improve the rigor and ultimately the benefits of federally funded scientific research writing:

The problem is that the research conducted using federal funds is driven — and distorted — by the academic publishing model. The intense competition for space in top journals creates strong pressures for novel, statistically significant effects. As a result, studies that do not turn out as planned or find no evidence of effects claimed in previous research often go unpublished, even though their findings can be important and informative.

The author contends although the scientific community has taken steps to improve research practices, the solutions have not adequately addressed the problem of publication bias:

One approach is to require researchers to share data, especially from studies conducted with public support. For instance, the National Institutes of Health and the National Science Foundation already require grantees to share data from their research. These sorts of requirements encourage transparency but, even if widely adopted, are unlikely to appreciably reduce bias in which studies are actually published.

Calling for an alternative solution, Nyhan places the onus for solving the issue of publication bias on academic journals and other institutions responsible for the publishing incentives faced by researchers:

Instead, my colleagues and I propose a radically different publishing model: Ask journal editors and scientific peers to review study designs and analysis plans and commit to publish the results if the study is conducted and reported in a professional manner (which will be ensured by a second round of peer review).

This procedure encourages authors and reviewers to develop the strongest possible designs — including those that replicate previously published studies  and eliminates perverse incentives to find or emphasize significant results after the fact. A new scientific format called Registered Reports using this approach has already been adopted at numerous journals across the social and natural sciences.

In a new white paper, I propose that the American Political Science Association offer options for articles in a Registered Reports-style format. Researchers in other academic disciplines and scientific associations are starting to do the same.

Nyhan concludes:

Scientists would change their ways much more rapidly if federal funding encouraged publishing in journals that used Registered Reports or other formats intended to minimize publication bias. Conversely, journals would be more likely to change their policies if it would help them attract research from top scientists. Appropriately enough, the best way to encourage scientific innovation might be to rethink how we organize the scientific enterprise itself.

Visit the Upshot’s website for the full article.