Guest post by W. Robert Reed, Professor in Economics in the Department of Economics and Finance, University of Canterbury, New Zealand and co-organizer of The Replication Network

Replication is hot. At least I think it must be, because typing “replication AND science” in Google produces hundreds of recent news stories in non-academic, or quasi-academic, media (e.g., here, here, and here). In addition, over the last few years, a number of organizations and websites have sprung up to promote research transparency and replication. BITSS is obviously a key player here. As is the Replication Wiki, 3ie’s replication programme, Curate Science, Political Science Replication, and others. Included in these is The Replication Network (TRN), which was formed in May of 2015.

The goal of TRN is to “to encourage economists and their journals to publish replications.” While organizations such as BITSS encourage research transparency broadly, and websites like the Replication Wiki collect information on replication studies, TRN’s focus is on getting more replication studies published in economics journals.

How important is it that journals publish replications? Imagine a world where every published study has its data and code provided, but journals don’t publish replication studies. What happens then? Individual researchers confirm or disconfirm research findings. But the discipline as a whole does not advance because invalid or nonrobust results are not publicly identified and discounted from the corpus of published research.

So how are things on the publishing front when it comes to replications? Not good. In a recent survey that I did with Maren Duvendack and Richard Palmer-Jones, we contacted all 333 economics journals listed in Web of Science to determine how many of them had a publicly stated policy of publishing replications. At the time of the survey (about a year ago), the answer was 10.

At one level, it’s even worse than that number suggests. Some of these journals, such as Experimental Economics and Economics of Education Review, only publish replications in their specialty areas. Others, such as the Journal of Applied Econometrics, only publish replications where the original article was published in one of a few elite journals. While it is perfectly appropriate that journals limit their publishing to specific discipline areas, as a practical matter, it means that there may only be one or two journals that are feasible publishing outlets for a researcher’s replication study.

All is not doom and gloom. Some journals, like Public Finance Review, are not listed in Web of Science but have active replication sections. And Economics: the Open Access, Open Assessment e-Journal, is listed in WOS and recently opened up a replication section.

As a follow-on to our survey, TRN provides an up-to-date listing of economics journals that publish replications (see here). Some journals publish replications without explicitly stating so. The American Economic Review is an example. As a result, TRN also provides an up-to-date spreadsheet of published replication studies that is searchable by journal, year of publication and author (see here). These resources can help researchers identify outlets for their replication research.

Overall, though, it is fair to say that the opportunities for economists wishing to publish replication studies in respectable journals are currently fairly limited. So how can one fix this?

One way is to let journals, and their editors, know that there is a large market of researchers who are keen to see replications published. Towards this end, TRN asks researchers to join as members. Currently, TRN has over 200 members (see here). We aim to eventually get to 1000+ members. The hope is that a publicly-displayed list of members that is both high in quantity and quality will impress journals and their editors that there is demand for this kind of research.

The above addresses how TRN attempts to “encourage journals to publish replications.” The other side of the equation is how to encourage economists to undertake more replication studies. One of the other hats that I wear is as replications editor for Public Finance Review. PFR opened up a replications section in 2010. One of the surprising things to me is how few replication studies have been submitted to the journal during that time period. I don’t think our experience is unique.

In a guest blog for TRN, Joop Hartog writes: “In 1994, my friend Jules Theeuwes and I launched the journal Labour Economics. We wanted a balanced mix of theory and empirical work and we opened a separate section for Replications, with a separate editor. As replication submissions were barely forthcoming (one paper in 3 years), we decided to beat the drum. … We guaranteed publication of replication studies, provided they would meet some mild conditions (aim to replicate key findings of an original article in a leading journal, and contain no methodological flaws). To our regret, we never got a single submission.”

So, while replications might seem hot, the reality is very different. A lot of talk, but not much action. How can one fix this? Great question. I’d love to hear some answers! TRN’s approach is to stoke enthusiasm for replications via an interesting provision of Guest Blogs and News & Events stories. The hope is that by collecting interesting and provocative articles along with useful resources all in one place, researchers will be encouraged to undertake replication research. Many embers can produce a lot of heat! That, at least, is the theory.

What do I hope to accomplish with this blog? I am hoping that you will join The Replication Network. Our “business model” works a little differently than other websites. While one can subscribe via RSS or follow us on Twitter, our main form of communication consists of an email update every few months in which we summarize the new content on TRN. That way you can see if there is anything worth checking out, while keeping your inbox from getting clogged up with notifications of each new posting.

There is no cost to joining TRN, and it is easy to unsubscribe. Further, membership at TRN is a nonrivalrous activity. It doesn’t preclude one from subscribing to the great content at BITSS or elsewhere. I also subscribe to BITSS!

To join TRN, go to this website and click where it asks if you are interested in becoming a member: http://replicationnetwork.com/membership/ .

Replication is no silver bullet. But it is a crucial piece of the scientific process that our discipline has largely ignored for too long. It doesn’t have to stay that way.