Frequently Asked Questions for Reviewers
A. Stage 1 of Peer Review
Why does the paper I was sent not have any results?
This paper was submitted in the Pre-Results Review track at the Journal of Development Economics. This submissions track involves a two-stage review process where prospective quantitative research projects are reviewed and accepted before the results are known. Throughout the publishing process, we refer to these articles as Registered Reports (RR), in line with the terminology adopted by over 100 academic journals and the existing Elsevier publishing infrastructure.
At Stage 1, authors submit a Proposal for a prospective empirical project, which typically contains a literature review, research question(s), hypotheses, and detailed methodological framework. Following Stage 1 peer review, high quality studies are accepted based on pre-results review, which constitutes a commitment by the JDE to later publish the full paper regardless of the nature of its results. After Stage 1, authors collect and analyze data, and then submit a full paper (including results and discussion sections) for Stage 2 review. This paper is published as long as the data collection and analysis maintain standards of quality and the study was implemented in alignment with the research design accepted at Stage 1.
Learn more in the JDE Registered Reports Author Guidelines and see the Flowchart of the RR Peer Review Process.
How should a Stage 1 referee report for a Registered Report submission differ from a standard referee report?
Referees should base their assessment of Stage 1 Proposals on the following criteria:
- The importance of the research question(s), in terms of its contribution or value added to the development economics literature;
- The logic, rationale, and plausibility of the proposed hypotheses;
- The soundness and feasibility of the methodology and statistical analysis plan (including statistical power analysis, where appropriate);
- Whether the clarity and degree of methodological detail are sufficient to replicate the proposed experimental procedures and statistical analysis plan in line with the JDE Mandatory Replication Policy;
- Whether the authors have pre-specified sufficient tests for ensuring that the results obtained can test the stated hypotheses, including data quality checks for data accuracy, consistency, bias, and completeness.
Proposals need to satisfy all of the criteria above. Based on the referee recommendations, Stage 1 Proposals will be 1) rejected; 2) returned to authors for the opportunity to revise and resubmit; or 3) accepted based on pre-results review. Learn more about Stage 1 of the review process in the JDE RR Author Guidelines.
If I see problems with the design of the study at Stage 1, should I recommend changes?
Referees are welcome to make recommendations that may help Proposals satisfy one or more of the Stage 1 review criteria and improve the quality of the resulting papers.
However, referees should note that in some cases Stage 1 Proposals are based around work that is well underway. If for example, an author has a sample and baseline data and/or the analysis is pre-registered, it may be problematic or quite costly for the author to construct a new randomization strategy, or to significantly deviate from his/her proposed analysis. Recommendations should be in the direction of ensuring that the proposed study meets the quality standards of the JDE, however may be disregarded if the proposed study already is of satisfactory quality. The Editorial Board will decide which changes are essential for the study to be accepted based on pre-results review, and which ones are discretionary for the author(s).
The Proposal is methodologically sound, but is of limited relevance to development economics. Can I recommend that it is rejected?
Yes. Successful Proposals should be positively evaluated based on all of the Stage 1 criteria. If referees find that the Stage 1 Proposal is not relevant to the field, or has a limited potential to make a contribution to the development economics literature, they are free to recommend changes or a rejection.
The study I was asked to review has already started data collection. How should I assess whether the author(s) looked at the data before submitting?
Empirical studies where data collection has already started at the time of Stage 1 review are eligible as long as authors can verifiably demonstrate that they had not yet looked at any data that will be used to test the pre-specified hypotheses. Below is a non-exhaustive list of possibilities for how authors can demonstrate this:
- Data collected through an intermediary – In cases where authors are involved in setting up data collection mechanisms, but rely on an intermediary to collect the data, they can ask the intermediary to restrict access to the data until the study is accepted based on pre-results review. In such cases, authors may support their Stage 1 Proposal with a signed statement, data use agreement with the intermediary, or other means of verification;
- Non-public data – When using non-publicly available administrative or survey data (e.g. data from the Demographic and Health Surveys Program), authors can request the owner of the data verify that they had not accessed the data prior to submitting their Stage 1 Proposal (e.g. a download receipt and signed statement affirming this was the first download of the data).
- Prospective administrative data – When proposing analyses using administrative datasets that have not yet been released, authors can submit their Stage 1 Proposal before the release date;
- Pre-registered pre-analysis plans – In cases where authors have collected and/or accessed the data at the time of submission, they can rely on a time-stamped registration that specified the research design in detail before the commencement of data collection. Such registration should be submitted as supporting material to the Stage 1 Proposal.
Referees can ask for additional assurances to demonstrate how the proposed analyses were not biased by the data, or recommend that the Stage 1 Proposal be rejected.
B. Stage 2 of Peer Review
How should a Stage 2 referee report for a Registered Report differ from a standard referee report?
Stage 2 of peer review in RRs ensures that data collection and analysis maintain high standards of quality and the study was implemented in alignment with the research design accepted at Stage 1.
Referees should base their assessment of Stage 2 Articles on the following criteria:
- Whether the study was implemented according to the pre-specified research design accepted at Stage 1. This involves an evaluation of whether the study implementation maintained high standard of quality, particularly whether the attrition rate was high in the given context, or there was a high differential attrition between treatment and control groups;
- Whether the collected data is of sufficient quality to test the pre-specified hypotheses. In particular, whether the pre-specified data quality checks were satisfied and descriptive statistics provide convincing evidence that the data could credibly measure the impact of the intervention;
- In cases of deviations from the pre-specified research design, whether the author(s) provided a convincing theoretical and/or methodological justification;
- Whether any exploratory analyses added by the authors are justified, methodologically sound, and informative. Exploratory analyses must be clearly noted, and the results of all hypotheses specified in the Stage 1 submission must be reported.
- Whether the author’s’ interpretation of the research findings are consistent with the data and the overall evidence.
Referee reports should provide an evaluation of the submission strictly based on the Stage 2 criteria above. Please also note that editorial decisions will not be based on the statistical significance, perceived importance, novelty or conclusiveness of the results of the study. This is a key feature of pre-results review. Reviewers at Stage 2 may suggest that authors report additional tests that were not pre-specified; however, authors are not obliged to complete these tests unless such tests are necessary to satisfy one or more of the Stage 2 review criteria.
Learn more about Stage 2 of the review process in the JDE RR Author Guidelines
Can I recommend that the paper is rejected at Stage 2 if I see problems with the research design?
Referees can assess the soundness and feasibility of the research design and make relevant suggestions only at Stage 1 of the peer review process. At Stage 2, the focus of the peer review process should switch to assessing the quality of implementation of the pre-specified research design based on the Stage 2 criteria.
Whereas referees are free to make comments regarding the pre-specified research design in their Stage 2 referee reports, these will be considered irrelevant at Stage 2, and will not influence editorial decisions.
What kinds of deviations from the research design accepted at Stage 1 are acceptable?
Authors are expected to implement the study based on the research design accepted at Stage 1. However, in the case of a deviation from the pre-specified design, the Editors will strive to err in favor of the authors, as long as the deviation(s) are reported transparently and justified based on methodological and/or theoretical considerations. Referees will evaluate whether the deviations compromise the integrity of the pre-specified research design and the credibility of the findings, and ultimately whether they find the author’s justification acceptable.
Please reference the ‘Deviations from the Pre-specified Research Design’ section in the JDE RR Author Guidelines for a non-exhaustive list of deviations that may render a study ineligible to be published as a Registered Report. However, this pilot will help the JDE identify other examples of unacceptable deviations. Referees are encouraged to make recommendations in this regard in their Stage 2 referee reports.
Can authors change or omit any of the pre-specified hypotheses in their Stage 2 submissions?
No. The results of all hypotheses pre-specified at Stage 1 must be included in the full manuscript submitted at Stage 2. In instances where a pre-specified hypothesis is subsequently shown to be logically flawed or unfounded, authors may include it in an Appendix (if particularly lengthy) or as a footnote, rather than in the ‘Results’ section.