By Aleksandar Bogdanoski, Katie Hoeberling, and Fernando Hoces de la Guardia

This post is also published on the CEGA blog.

The COVID-19 pandemic has arguably had a more profound impact on scientific research than any other phenomenon in recent history; between January and May this year, over 23,000 research papers were written on the topic. While this research is urgently needed, it is not immune to the issues that already threaten the credibility of the scientific enterprise, including publication bias, the “file drawer problem,” p-hacking, and limited sharing of data, code, and study materials. Though peer review in academic journals should ensure research quality, the sheer volume of research on COVID-19 makes this virtually impossible and stresses the need for transparency and opening up the quality vetting process to the scientific community.

Policy is only as effective as the evidence on which it rests. Transparent research practices can help us detect oversight, mistakes, conflicts of interest, and in the worst cases, fraud. Access to study data and pre-registrations has already helped the biomedical research community correct course on two controversial COVID-19 treatments (hydroxychloroquine and remdesivir), and it can have a similarly profound impact in social science research, as well. In such a time of need and prolific research production, the stakes are higher than ever to ensure scientific research on the pandemic is as accurate and credible as possible. By creating a public record of research findings, many of which remain unpublished by journals, the value of pre-registration — and particularly pre-analysis plans (PAPs) — cannot be understated.

Coordinating and streamlining the COVID-19 research agenda

Social scientists are working hard to help governments understand and mitigate the devastating socio-economic impacts of the pandemic. Yet this research is subject to the same concerns of coordination, credibility, and quality: those involved in its production cannot afford to duplicate efforts, nor does it serve anyone if its findings never see the light of day. Recognizing this, several organizations have stepped up with initiatives to coordinate the production and dissemination of COVID-19-related research. The Research for Effective COVID-19 Responses (RECOVR) hub, led by Innovations for Poverty Action, and the COVID-19 List of Projects curated by the European Economic Association, among others, are enabling a freer flow of ideas and information about ongoing and prospective work.

While such project registries provide information on authors, topics, and basic research design, the coordination they foster can be further improved if researchers also write PAPs and register them on conventional public hypothesis registries like the AEA RCT Registry, OSF, EGAP, or RIDIE before collecting or accessing data. Beyond providing basic information about research questions and designs, PAPs provide a richer context by clearly detailing planned hypotheses, identification strategies, sampling plans, variables, and statistical specifications. Making this information publicly available allows researchers to see not only what others are studying but how. Since we know there are a multitude of ways to test one hypothesis, broad access to the analysis plans of ongoing research could dramatically reduce the risk of duplication and make collaboration between researchers with related interests much easier.

PAPs can also facilitate ex-ante conversations about planned research, which are often much more helpful and constructive than those that happen once work has already completed (just ask Reviewer 2!). This is particularly true for tests of costly or complex interventions that will be difficult to replicate. Considering that at least $8 billion in new funding is needed for the global response to COVID-19, it is crucial that study designs are as rigorous as possible and avoid redundancy and duplication. Conferences and seminars, many of which are moving online, are useful venues for discussing PAPs about prospective work. Furthermore, well-elaborated PAPs have the chance of being granted in-principle journal acceptance at journals that accept registered reports, such as the Journal of Development Economics! Finally, pre-registration helps make research discoverable and leaves a more complete record of evidence, which is useful for synthesizing evidence (e.g. through a meta-analysis) and guiding policy.

Providing rigorous and credible evidence for policymaking

PAPs have been shown to reduce publication bias and related challenges like underreporting, undisclosed analytic flexibility, p-hacking, and HARK-ing. Think of pre-registering a PAP as a researcher’s promise to their future self (“[t]he first principle is that you must not fool yourself”) and to readers of their research that their reported findings are the result of confirmatory hypothesis-testing. The alternative scenario is one mired in hindsight bias, where researchers use data to generate explanations but present them as predictions. With the rapid release of so much data, there are plenty of opportunities to find post hoc explanations if researchers test dozens, or even hundreds, of outcomes. This raises the risk of producing research that is not replicable—a frightening prospect when this research is used to inform policy.

Furthermore, with politically charged debates surrounding proposed public health measures, pharmaceuticals, and economic policies to address COVID-19, implicit pressure from donors and policymakers—or the perception of their existence—may incentivize researchers to produce certain results. Donor agendas may be bolstered by evidence that a flagship program is effective, for example. PAPs and pre-registration can help shield off such pressures, particularly when findings may be “inconvenient” or the payout for tendentious reporting is high.

And while global containment measures have made it difficult for researchers to collect experimental data as they would have before such measures were put in place, pre-registration can be applied to observational work. Researchers can use their familiarity with the structure of their own survey datasets  by pre-registering a PAP before collecting data. They can pre-register prospective analyses of datasets that will be released in the future, such as prospective census data or COVID-19 case data. They can pre-register before a policy change occurs (future stages of California’s Stay-at-Home orders, for example). Or they can use a time-stamped request for access to restricted-access datasets to credibly pre-register designs before obtaining access. Even for observational work that can’t be credibly pre-registered, the practice still has important benefits for coordinating scientific research, making the totality of evidence publicly available, and facilitating evidence synthesis.

Towards a more efficient and credible research process

Transparent practices like pre-registration and the use of PAPs, in addition to sharing data, code, materials, and findings, can help the social science community coordinate COVID-19 research more efficiently, and ensure that evidence generated during this unprecedented time is as credible and useful as possible. If you’re a researcher and not sure where to start, head to the BITSS Resource Library where you’ll find helpful teaching resources like guides and templates on all things pre-registration and PAPs, or to the Center for Open Science’s guide to pre-registration.

Many questions remain regarding how to effectively contain and mitigate the spread of the novel coronavirus, and the rapid proliferation of research is sure to continue. Coordination and transparency in research will be crucial in informing successful policy responses.