Accelerating Computational Reproducibility in Economics (ACRE)

Replication, robustness checks, and extension of published work are possible to the extent that published findings are first computationally reproducible. In other words, one must be able to reproduce tables and figures within a reasonable margin of error using the available data, code, and materials. In July 2019, to improve computational reproducibility in economics, the American Economic Association (AEA) updated their Data and Code Availability Policy to include pre-publication verification of computational reproducibility by the AEA Data Editor. 

In collaboration with the current AEA Data Editor Lars Vilhuber, BITSS launched the Accelerating Computational Reproducibility in Economics (ACRE) project. ACRE builds capacity for social science researchers and is developing the Social Science Reproduction platform, an online tool for systematically sourcing and recording the results of attempts to verify the computational reproducibility of published work.

Training

ACRE supports social science researchers in meeting journal editorial expectations of pre-publication verification of computational reproducibility. Capacity-building activities include:

  • Training events for participants of major association meetings in economics (e.g., Allied Social Sciences Association, the Association for Public Policy Analysis and Management, the Western Economics Association International, etc.). Learn about past and upcoming ACRE training events here.
  • Guidance resources for improved workflow reproducibility for AEA authors.

Social Science Reproduction Platform

The SSRP will facilitate the systematic sourcing, cataloging, and review of attempts to verify and improve the computational reproducibility of published work in economics. The platform will allow users to:

  • Record the results of reproduction attempts through a standardized form;
  • Review, comment, and collaborate on reproduction attempts submitted by other users on the platform; and
  • Access aggregate metrics of reproducibility across papers, journals, sub-disciplines, timespans, etc.
The SSRP facilitates steps 0 through 4.

The SSRP is based on the Guidelines for Accelerating Computational Reproducibility, collaboratively developed by BITSS and members of the open social science community. The Guidelines contain detailed steps and criteria for assessing and improving computational reproducibility, as well as resources for constructive and efficient communication between reproducers and original authors. The Guidelines and SSRP can be used as a curricular module for reproductions performed in graduate and undergraduate economics courses. We estimate that graduate students will spend 10-15 hours reproducing a single finding from analytic data.

The SSRP is currently under development and will be widely available in Winter 2021.

Contact

Reach out to acre@berkeley.edu if you are interested in using the ACRE curricular module as part of your course.