Frequently Asked Questions for Authors

A. Features of the Registered Reports (RR) format

How are Registered Reports different from study pre-registration and pre-analysis plans (PAPs)?

Pre-specifying the details of a research design before data collection and analysis is central to the RRs publishing format, pre-analysis plans (PAPs), and study pre-registration. What distinguishes the RR format from the other two is that it also incorporates peer-review of the pre-specified research design at Stage 1. Following data collection and analysis, Stage 2 of peer review then ensures that research projects were implemented in alignment with their pre-specified research design, and the results of all pre-specified analyses are clearly reported.

If accepted based on pre-results review, can authors first submit the full paper to another journal?

Following acceptance based on pre-results review authors are free to first submit the full-length paper to other journals. This will not compromise the Stage 2 review of the paper if it is subsequently submitted to the JDE.

Note that the Stage 1 proposal or the Stage 2 full paper must not be submitted to another journal while the respective manuscript is under review at the JDE. The Editorial Board therefore asks that corresponding authors notify the JDE if they decide to submit the full paper to another journal. Should the paper be rejected by the other journal, the JDE will observe the pre-results acceptance for up to five years after it was granted.

If the full paper is accepted at another journal, the corresponding author should notify the JDE, and a brief note on the outcome will be published alongside the proposal’s summary on the JDE RRs page on the BITSS website (e.g., “The proposal <title> by <authors> was accepted by the Journal of Development Economics based on pre-results review basis. The full paper based on this proposal was eventually accepted and published as <title> <citation>,<DOI>). The JDE Editorial Board also asks that working papers and published papers acknowledge that the article underwent peer review and was accepted based on pre-results basis at the JDE, citing the proposal summary on the JDE RRs page on the BITSS website.

Are Stage 1 Proposals required to be pre-registered?

The JDE does not require, but strongly encourages authors to pre-register their Stage 1 Proposals before the start of data collection. Authors can use the information from their Stage 1 Proposal submission to easily pre-register their study.

For more information, please consult the ‘Pre-registration Resources’ section in the JDE RR Author Guidelines.

Does the RR format limit exploratory research?

The RR format does not restrict authors’ ability to conduct post hoc, exploratory analyses and report them in the full-length article submitted at Stage 2. However, such analyses must be clearly labeled as exploratory and distinguished from the confirmatory analyses that were specified prior to data collection and analyses.

Will the JDE publish Stage 1 Proposals that have been accepted based on pre-results review?

Stage 1 Proposals will not be published as stand-alone items in the JDE following acceptance based on pre-results review. BITSS will post the title and abstract of Stage 1 Proposals that were accepted based on pre-results review in the “Prospective Articles” page on the BITSS website, unless authors choose not to have them posted.

Note that authors are required to submit the accepted Stage 1 Proposal as supporting material to their Stage 2 submissions. Stage 1 Proposals will be included in the appendix of the published article.

Publicizing an ongoing study once the Proposal is accepted may bias the behavior of study participants. Is it possible to still post the title and abstract of the Proposal, while masking sensitive information?

Yes. Authors should communicate such concerns to the Editors and BITSS, and point out the details that need to be masked.

How will the JDE ensure that papers in the RR are based on high quality data if acceptance is granted before data are available?

During Stage 1 peer review and before data is collected, reviewers will assess whether authors have pre-specified sufficient data quality checks for accuracy, consistency, bias, and completeness. At Stage 2, reviewers will verify that pre-specified data quality checks have been met, and that data is of sufficient quality to test the pre-specified hypotheses.

In addition to pre-specified data quality checks, as part of the Stage 1 RR submission, authors are advised to consider potential challenges that may arise during implementation (e.g., attrition, non-compliance with treatment) and offer strategies to address them.

B. Eligibility of Study Design and Data

Are only randomized controlled trials (RCTs) eligible for the RR submissions track?

No, this format is not limited to any particular study design, but may be particularly well suited for RCTs. Any study design may be eligible if the data are yet to be collected/accessed and analyzed at the time of submission. This also includes quasi-experimental and observational studies (e.g., Neumark (2001), Burlig (2018), and the 2016 Election Research Pre-acceptance Competition).

At what stage of data collection in the project should the Stage 1 Proposal be submitted? Are projects where some data have been collected eligible for this track?

As of January 1, 2019, submissions where follow-up data have been collected (outside of pilot and baseline data) are no longer eligible for this track.

Work should be submitted at a minimum of three months, but ideally six months to a year, before data collection of the key outcome data. All participants, authors, referees and editors, must be blind to the key outcome variables at each stage of revision of the Stage 1 submission. This is useful for two reasons: (a) to maximize the possibility that feedback can be used in shaping final data collection, and (b) to ensure enough time for an R&R round and re-submission before the final data come in.

(Note: it is typically quite useful to have baseline data in hand when submitting the Stage 1 Proposal for pre-results review: baseline data provides useful information on the nature of measures, including their means and variances, which can help to understand the context and carry out meaningful statistical power calculations.)

Are pilot data required for Stage 1 RR submissions?

No, however authors may include pilot data in their Stage 1 submissions to establish proof of concept, effect size estimations, and/or feasibility of proposed methods.

Are prospective research projects based on split-sample techniques eligible?

In this approach, researchers use a training sample to run preliminary analyses that inform the Stage 1 Proposal and then refrain from analyzing the rest of the sample. After the Proposal is accepted, researchers use the remainder of the sample (testing sample) to test the pre-specified hypotheses, which constitute the main results of the published study.

Yes, this type of study design is eligible. However, at the time of the Stage 1 submission, authors will need to verifiably demonstrate they have not seen the testing part of the sample before the submission date (e.g., a download receipt plus a signed statement affirming this was the first download of the data).

C. Preparing Submissions

Can authors submit their pre-analysis plans or pre-registrations as a Stage 1 Proposal to the JDE?

PAPs and/or pre-registrations can serve as a helpful base for your Stage 1 submission. Stage 1 Proposals go a step further and require that authors outline the importance the research question in terms of its contribution or value added to the development economics literature, as in a regular full-length submission (see criterion 1 of the Stage 1 review criteria here). Stage 1 Proposals are expected to mirror full-length conventional articles, with the exception of the ‘Results’ and ‘Discussion’ sections. Authors may refer to the optional Stage 1 RR Submission Template for more guidance.

How should Stage 1 Proposal submissions differ from standard JDE articles?

Stage 1 Proposal submissions should follow the same structure and formatting rules as standard JDE article submissions, however need to provide enough details to allow for the project to be evaluated in the light of Stage 1 assessment criteria (learn more in the JDE RR Author Guidelines). For items of the Stage 1 Proposal that are not commonly found in JDE papers (e.g. ex-ante plans for data processing, variations from intended sample size, pilot data, etc.), authors can include brief summaries in the body of the paper, and include the rest as an appendix to the Stage 1 Proposal submission.

Some studies involve multiple experiments where the later experiments are contingent on the results of earlier experiments. How can authors pre-specify the full research design in the Stage 1 submission?

In studies where parts of an analysis depend on the results of another, authors may consider constructing a decision tree as part of their Stage 1 Proposal (e.g., “If A is observed, then we will adopt analysis A1 but if B is observed then we will adopt analysis B1”). Alternatively, a single primary outcome (or a narrow set of outcomes) can be pre-specified at Stage 1, and the remainder of the analyses may be conducted as exploratory.

Is IRB (ethics) approval required at the time of submitting the Stage 1 Proposal?

Yes, however exceptions may be granted on a case-to-case basis. The JDE will not publish the resulting full-length article unless it has successfully obtained IRB approval.

How should Stage 2 Article submissions differ from standard JDE articles?

Publications in this track will look like any other article published in the JDE, with two exceptions: i) the main text will contain a footnote stating that the paper was accepted based on pre-results review, and ii) the accepted Stage 1 Proposal will be included in the supplementary online appendix.

Here are some additional tips to consider:

  • Authors must report deviations from the pre-specified research design. These can be addressed throughout the body of the Stage 2 full-length article, or in a specific “deviations from the from pre-specified research design” section of the paper, along with details on the context. If particularly lengthy, they may be included in the appendix.
  • Authors are expected to report the results of all hypotheses from the Stage 1 Proposal in the Stage 2 Article, unless a hypothesis is proven to be logically flawed or unfounded. In such cases, it can be included in the appendix with a description and/or justification.
  • Analyses that were not part of the Stage 1 submission should be appropriately labeled (e.g. as exploratory, additional, post-hoc, etc.), and included in a separate section.
How can authors submit a full paper for Stage 2 review?

Once a Stage 1 Proposal is accepted based on a pre-results review, authors will receive an official notice from the Editors.

Note that due to functionality limitations of our manuscript handling platform, this will be paired with an auto-generated “revise and resubmit” notification, though this is not the actual decision. To submit a full paper for Stage 2 review, authors should  submit it by responding to the “revise and resubmit” notification, and choosing the article type from Registered Report Stage 1: Proposal to Registered Report Stage 2: Full Length Article.

D. Deviations from the Research Design Specified in the Stage 1 Proposal

Can authors change or omit any of the pre-specified hypotheses after acceptance based on pre-results review?

No. The results of all hypotheses pre-specified at Stage 1 must be included in the full manuscript submitted at Stage 2. In instances where a pre-specified hypothesis is subsequently shown to be logically flawed or unfounded, authors may include it in an Appendix (if particularly lengthy) or as a footnote, rather than in the ‘Results’ section.

Can authors conduct other analyses in the Full Article in addition to those pre-specified in the Proposal that was accepted at Stage 1?

Yes, as long as such analyses are clearly caveated (e.g. reported in a separate section) from the analyses based on the Proposal that was accepted by the JDE at Stage 1.

This may be relevant in instances where the author(s) submit a Stage 1 Proposal based on a pre-analysis plan (PAP) for a project in collaboration with an external partner (e.g., government, funding agency) that asks for particular empirical analyses (e.g., estimates over sub-sample A, given primary outcome(s), etc.), however underwent revision during Stage 1 of peer review. If partners require the original PAP as is, then those analyses can be presented in the final paper, but in addition to the analyses accepted by the JDE at Stage 1.

Due to substantial deviations from the pre-specified research design, some studies may no longer be eligible to be published as Registered Reports. Can those still be submitted as full-length articles to the JDE?

Yes. Such articles will be considered in the standard review process.

E. Editorial Triage and Peer Review Process

What are the possible outcomes of Stage 1 peer review?

During Stage 1, submissions will be either 1) rejected; 2) returned to authors for the opportunity to revise and resubmit; or 3) accepted based on pre-results review. Pre-results acceptance constitutes a commitment by the JDE to later publish the resulting full paper regardless of the nature of its empirical findings, subject to a successful Stage 2 review.

How long will the review process be for articles submitted in the RR track?

Information about the duration of peer review for full-length papers in the standard peer review track at the JDE is available here. Given the novelty of the RR track, the JDE does not have information on the duration of the two-stage peer review process yet, however the Editorial Board will make an effort to streamline the process as much as possible.

Once offered acceptance based on pre-results review, authors can include the article as an upcoming publication in their research portfolios with the label: “Accepted at the Journal of Development Economics based on pre-results review”, and the URL of the proposal summary in the “Prospective Articles” page on the BITSS website.

How long will the JDE’s pilot phase last? If a study is accepted on pre-results review basis and completed beyond that period, will the JDE still publish the paper?

We do not currently have an expiration date for this pilot. However, the JDE will honor its commitment to publish papers that successfully underwent Stage 1 review, regardless of whether RRs are retained as a publishing model at the time of publication.

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.