Accelerating Computational Reproducibility (ACRe)
Replication, robustness checks, and extension of published work are possible to the extent that published findings are first computationally reproducible. In other words, one must be able to reproduce tables and figures within a reasonable margin of error using the available data, code, and materials. In July 2019, to improve computational reproducibility in economics, the American Economic Association (AEA) updated its Data and Code Availability Policy to include pre-publication verification of computational reproducibility by the AEA Data Editor. Similar policies have been adopted in political science, particularly at the American Journal of Political Science.
In collaboration with the current AEA Data Editor Lars Vilhuber, BITSS launched the Accelerating Computational Reproducibility (ACRe) project. ACRE builds capacity for social science researchers and is developing the Social Science Reproduction platform, an online tool for systematically sourcing and recording the results of attempts to verify the computational reproducibility of published work.
ACRE supports social science researchers in meeting journal editorial expectations of pre-publication verification of computational reproducibility. Capacity-building activities include:
- Training events for participants of major association meetings in economics (e.g., Allied Social Sciences Association, the Association for Public Policy Analysis and Management, the Western Economics Association International, etc.). Learn about past and upcoming ACRE training events here.
- Guidance resources for improved workflow reproducibility for AEA authors.
The Social Science Reproduction Platform (SSRP) (now in beta) is an open-source platform that crowdsources and catalogs attempts to assess and improve the computational reproducibility of published social science research. The platform allows users to:
- Record the results of reproduction attempts through a standardized form;
- Review, comment, and collaborate on reproduction attempts submitted by other users on the platform; and
- Access aggregate metrics of reproducibility across papers, journals, sub-disciplines, timespans, etc.
The SSRP is based on the Guide for Accelerating Computational Reproducibility (ACRe Guide), collaboratively developed by BITSS and members of the open social science community. The ACRe Guide contains detailed steps and criteria for assessing and improving computational reproducibility, as well as resources for constructive and efficient communication between reproducers and original authors. The ACRe Guide and the SSRP can be used as a curricular module for reproductions performed in graduate and undergraduate economics courses. We estimate that graduate students will spend 10-15 hours reproducing a single finding from analytic data.
Reach out to email@example.com if you are interested in using the ACRE curricular module as part of your course or would like to contribute to the ACRe Guide or the SSRP.