Frequently Asked Questions for Authors

A. Features of the Registered Reports (RR) format

How are Registered Reports different from study pre-registration and pre-analysis plans (PAPs)?

Pre-specifying the details of a research design before data collection and analysis is central to the RRs publishing format, pre-analysis plans (PAPs), and study pre-registration. What distinguishes the RR format from the other two is that it also incorporates peer-review of the pre-specified research design at Stage 1. Following data collection and analysis, Stage 2 of peer review then ensures that research projects were implemented in alignment with their pre-specified research design, and the results of all pre-specified analyses are clearly reported.

If accepted based on pre-results review, do authors have to submit the full paper to the JDE?

Following acceptance based on pre-results review authors are free to first submit the full-length paper to other journals. This will not compromise the Stage 2 review of the paper if it is subsequently submitted to the JDE.

Note that the Stage 1 proposal or the Stage 2 full paper should not be submitted to another journal while the respective manuscript is under review at the JDE. The Editorial Board therefore asks that corresponding authors notify the JDE if they decide to submit the full paper to another journal.

If the full paper is accepted at another journal, the corresponding author should notify the JDE, and a brief note on the outcome will be published alongside the proposal’s summary on the JDE RRs page on the BITSS website (e.g., “The proposal <title> by <authors> was accepted by the Journal of Development Economics based on pre-results review basis. The full paper based on this proposal was eventually accepted and published as <title> <citation>,<DOI>). The JDE Editorial Board also asks that working papers and published papers acknowledge that the article underwent peer review and was accepted based on pre-results basis at the JDE, citing the proposal summary on the JDE RRs page on the BITSS website.

Are Stage 1 Proposals required to be pre-registered?

The JDE does not require, but strongly encourages authors to pre-register their Stage 1 Proposals before the start of data collection. Authors can use the information from their Stage 1 Proposal submission to easily pre-register their study.

For more information, please consult the ‘Pre-registration Resources’ section in the JDE RR Author Guidelines.

Does the RR format limit exploratory research?

The RR format does not restrict authors’ ability to conduct post hoc, exploratory analyses and report them in the full-length article submitted at Stage 2. However, such analyses must be clearly labeled as exploratory and distinguished from the confirmatory analyses that were specified prior to data collection and analyses.

Will the JDE publish Stage 1 Proposals that have been accepted based on pre-results review?

Stage 1 Proposals will not be published as stand-alone items in the JDE following acceptance based on pre-results review. BITSS will post the title and abstract of Stage 1 Proposals that were accepted based on pre-results review in the “Prospective Articles” page on the BITSS website, unless authors choose not to have them posted.

Note that authors are required to submit the accepted Stage 1 Proposal as supporting material to their Stage 2 submissions. Stage 1 Proposals will be included in the appendix of the published article.

How will the JDE ensure that papers in the RR are based on high quality data if acceptance is granted before data are available?

During Stage 1 peer review and before data is collected, reviewers will assess whether authors have pre-specified sufficient data quality checks for accuracy, consistency, bias, and completeness. At Stage 2, reviewers will verify that pre-specified data quality checks have been met, and that data is of sufficient quality to test the pre-specified hypotheses.

In addition to pre-specified data quality checks, as part of the Stage 1 RR submission, authors are advised to consider potential challenges that may arise during implementation (e.g., attrition, non-compliance with treatment) and offer strategies to address them.

B. Eligibility of Study Design and Data

Are only randomized controlled trials (RCTs) eligible for the RR submissions track?

No, this format is not limited to any particular study design, but may be particularly well suited for RCTs. Any study design may be eligible if the data are yet to be collected/accessed and analyzed at the time of submission. This also includes quasi-experimental and observational studies (e.g., Neumark (2001), Burlig (2018), and the 2016 Election Research Pre-acceptance Competition).

Are studies where part of the data have been collected, but not yet analyzed, eligible for the RR submission track?

Such submissions are eligible as long as authors can verifiably demonstrate that they have not yet looked at any data which will be used to test the pre-specified hypotheses. This may not be possible with pre-existing widely available datasets.

In the Stage 1 Proposal, authors should clearly outline in as much detail as possible which data have been collected and how the proposed analyses remain blinded/unbiased by the results. If the proposed study was pre-registered and/or authors prepared a PAP before collecting data, these materials should be included in support of the Stage 1 RR submission (learn more below).

Are pilot data required for Stage 1 RR submissions?

No, however authors may include pilot data in their Stage 1 submissions to establish proof of concept, effect size estimations, and/or feasibility of proposed methods.

Are prospective research projects based on split-sample techniques eligible?

In this approach, researchers use a training sample to run preliminary analyses that inform the Stage 1 Proposal and then refrain from analyzing the rest of the sample. After the Proposal is accepted, researchers use the remainder of the sample (testing sample) to test the pre-specified hypotheses, which constitute the main results of the published study.

Yes, this type of study design is eligible. However, at the time of the Stage 1 submission, authors will need to verifiably demonstrate they have not seen the testing part of the sample (learn more below).

How can authors prove that they had not accessed the data before submitting the Stage 1 Proposal?

Authors are free to propose any means of verification that will demonstrate that the data used to test the pre-specified hypotheses were not accessed before the submission of a Stage 1 Proposal, except for purposes of checking during data collection. This is certainly not an exhaustive list, but here are a few possibilities:

  • Data collected through an intermediary – In cases where authors are involved in setting up data collection mechanisms, but rely on an intermediary to collect the data, they can ask the data collection intermediary to restrict their access to the data up to the point when the study is accepted based on pre-results review. In such cases, authors may support their Stage 1 Proposal with a signed statement, data use agreement with the intermediary, or other means of verification;
  • Non-public data – When using non-publicly available administrative or survey data (e.g., data from the Demographic and Health Surveys Program), authors can arrange for the owner of the data to verify that they had not accessed the data before the submission date (e.g., a download receipt plus a signed statement affirming this was the first download of the data).
  • Prospective administrative data – When proposing analyses using administrative datasets that have not yet been released, authors can submit their Stage 1 Proposal before the release date;
  • Pre-registered pre-analysis plans – Authors can rely on a time-stamped registration that specified the research design in detail before the commencement of data collection. The registration should be submitted as supporting material to the Stage 1 Proposal.

C. Preparing Submissions

Can authors submit their pre-analysis plans or pre-registrations as a Stage 1 Proposal to the JDE?

PAPs and/or pre-registrations can serve as a helpful base for your Stage 1 submission. Stage 1 Proposals go a step further and require that authors outline the importance the research question in terms of its contribution or value added to the development economics literature, as in a regular full-length submission (see criterion 1 of the Stage 1 review criteria here). Stage 1 Proposals are expected to mirror full-length conventional articles, with the exception of the ‘Results’ and ‘Discussion’ sections. Authors may refer to the optional Stage 1 RR Submission Template for more guidance.

Some studies involve multiple experiments where the later experiments are contingent on the results of earlier experiments. How can authors pre-specify the full research design in the Stage 1 submission?

In studies where parts of an analysis depend on the results of another, authors may consider constructing a decision tree as part of their Stage 1 Proposal (e.g., “If A is observed, then we will adopt analysis A1 but if B is observed then we will adopt analysis B1”). Alternatively, a single primary outcome (or a narrow set of outcomes) can be pre-specified at Stage 1, and the remainder of the analyses may be conducted as exploratory.

Is IRB (ethics) approval required at the time of submitting the Stage 1 Proposal?

Yes, however exceptions may be granted on a case-to-case basis. The JDE will not publish the resulting full-length article unless it has successfully obtained IRB approval.

D. Deviations from the Research Design Specified in the Stage 1 Proposal

Can authors change or omit any of the pre-specified hypotheses after acceptance based on pre-results review?

No. The results of all hypotheses pre-specified at Stage 1 must be included in the full manuscript submitted at Stage 2. In instances where a pre-specified hypothesis is subsequently shown to be logically flawed or unfounded, authors may include it in an Appendix (if particularly lengthy) or as a footnote, rather than in the ‘Results’ section.

Can authors conduct other analyses in the Full Article in addition to those pre-specified in the Proposal that was accepted at Stage 1?

Yes, as long as such analyses are clearly caveated (e.g. reported in a separate section) from the analyses based on the Proposal that was accepted by the JDE at Stage 1.

This may be relevant in instances where the author(s) submit a Stage 1 Proposal based on a pre-analysis plan (PAP) for a project in collaboration with an external partner (e.g., government, funding agency) that asks for particular empirical analyses (e.g., estimates over sub-sample A, given primary outcome(s), etc.), however underwent revision during Stage 1 of peer review. If partners require the original PAP as is, then those analyses can be presented in the final paper, but in addition to the analyses accepted by the JDE at Stage 1.

Due to substantial deviations from the pre-specified research design, some studies may no longer be eligible to be published as Registered Reports. Can those still be submitted as full-length articles to the JDE?

Yes. Such articles will be considered in the standard review process.

E. Editorial Triage and Peer Review Process

What are the possible outcomes of Stage 1 peer review?

During Stage 1, submissions will be either 1) rejected; 2) returned to authors for the opportunity to revise and resubmit; or 3) accepted based on pre-results review. Pre-results acceptance constitutes a commitment by the JDE to later publish the resulting full paper regardless of the nature of its empirical findings, subject to a successful Stage 2 review.

How long will the review process be for articles submitted in the RR track?

Information about the duration of peer review for full-length papers in the standard peer review track at the JDE is available here. Given the novelty of the RR track, the JDE does not have information on the duration of the two-stage peer review process yet, however the Editorial Board will make an effort to streamline the process as much as possible.

Once offered acceptance based on pre-results review, authors can include the article as an upcoming publication in their research portfolios with the label: “Accepted at the Journal of Development Economics based on pre-results review”, and the URL of the proposal summary in the “Prospective Articles” page on the BITSS website.

How long will the JDE’s pilot phase last? If a study is accepted on pre-results review basis and completed beyond that period, will the JDE still publish the paper?

We do not currently have an expiration date for this pilot. However, the JDE will honor its commitment to publish papers that successfully underwent Stage 1 review, regardless of whether RRs are retained as a publishing model at the time of publication.

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.