Can Greater Transparency Lead to Better Social Science?

In a recent article on the Monkey Cage, professors Mike FindleyNathan JensenEdmund Malesky and Tom Pepinsky  discuss publication bias, the “file drawer problem” and how a special issue of the journal Comparative Political Studies will help address these problems. 

Similar to a recent article by Brendan Nyhan, reposted on the BITSS blog, the university professors writing the article assert:

[S]cholars may think strategically about what editors will want […] this means that “boring” findings, or findings that fail to support an author’s preferred hypotheses, are unlikely to be published — the so-called “file drawer problem.” More perniciously, it can incentivize scholars to hide known problems in their research or even encourage outright fraud, as evinced by the recent cases of psychologist Diederik Stapel and acoustician Peter Chen.

To address these problems, the authors of the article have worked with the journal for Comparative Political Studies to release a special edition in which:

[A]uthors will submit manuscripts with all mention of the results eliminated […] Other authors will submit manuscripts with full descriptions of research projects that have yet to be executed […] In both cases, reviewers and editors must judge manuscripts solely on the coherence of their theories, the quality of their design, the appropriateness of their empirical methods, and the importance of their research question.

They claim:

In an ideal world, embracing transparency in this fashion will eliminate the pernicious incentives that we identified above. Authors should have no incentives to manipulate their results (after all, if you have a guarantee that your findings will be published anyway, what does it matter what you find?)

Although the professors disagree with much of the skepticism surrounding their work they recognize its limitations:

[W]e suspect that our special issue will attract mostly quantitative research on well-defined issues in political science. Exploratory, qualitative, historical, and case-based research is much harder to present in a results-free manner, and perhaps impossible to pre-register.

We also recognize that pre-registration may commit scholars to executing research plans that they quickly learn to be infeasible when implementing them.

Despite these issues the authors believe it is important to experiment with new ideas and assure readers they will:

[D]escribe as fully and transparently as possible all of the challenges [they] face, expected and unexpected, in implementing these procedures.

Find the full article here.