Press release: BERKELEY, CA (Wednesday, March 13, 2019) — BITSS collaborated with UC Berkeley economists Emmanuel Saez and Gabriel Zucman to produce a fully reproducible version of their policy report for Sen. Elizabeth Warren’s wealth tax plan.
Senator Elizabeth Warren declared her 2020 presidential bid on a platform based on policies to rebuild the American middle-class. A key part of her campaign will be a progressive wealth tax on the net worth of American households, including a 2% tax on net worth above $50 million, and an additional 1% tax on net worth above $1 billion. UC Berkeley economists Emmanuel Saez and Gabriel Zucman—the researchers whose analysis forms the basis of the plan—estimate that “75,000 American households (less than 0.1%) would be liable for the wealth tax and that the tax would raise around $2.75 trillion” in the first ten years of its implementation.
Such sweeping policy ideas are regularly the subject of contentious debates, such as those around raising the minimum wage, taxing carbon, or instituting universal healthcare. Arguments in such debates are often split along ideological lines, with opposing sides citing diverging estimates, assumptions, and data. Further, some arguments are too general to be useful, like the critique by the Economist that described Saez and Zucman’s analysis as based on assumptions that are “probably too rosy.” Disclosing in full the assumptions, data, and methods behind each analysis can reduce the risk of conflating ideology with facts, and allow others to critically assess the merits of the alternative.
“[Open Policy Analysis] can help policy debates transcend ideological differences, and instead focus on the technical merit of the proposed policy.”
That is why Saez and Zucman teamed up with BITSS to “open up” their analysis, publicly sharing all their data and methods to make it possible for anyone to assess its validity and credibility. Having worked with researchers in academia to tackle the credibility crisis in the social sciences by improving the transparency and reproducibility of research, BITSS is now looking to apply relevant lessons to policy analysis. The approach is called Open Policy Analysis (OPA), where data, code, materials, and clear accounts of methodological decisions are made freely available to facilitate collaboration, discussion, and reuse. BITSS Project Scientist Fernando Hoces de la Guardia, who leads the OPA initiative, argues that it can help policy debates transcend ideological differences, and instead focus on the technical merit of the proposed policy.
Applying OPA to Saez and Zucman’s analysis, the team developed an interactive plot to simulate the results of the analysis for different parameters (developed with the help of Sara Stoudt, a Berkeley Statistics PhD student and BIDS Fellow); an ‘open policy report‘ that details all data, code, and assumptions behind the analysis; and an open online repository that stores all materials necessary to reproduce the analysis from scratch. Using the interactive plot, readers can alter the number of tax brackets, apply different rates to each, and even adjust the estimated tax evasion rate, observing how each alteration affects the tax burden, tax base and projected revenue. The open policy report then goes a step further, and details how Saez and Zucman constructed each parameter, spelling out each assumption behind the proposal, including their estimated tax evasion rate, one of the proposal’s most contentious elements. Finally, readers can download all materials from the online repository, and recreate the analysis in full with minimal effort.
“To truly achieve the ideals of evidence-based policy, both policymakers and policy analysts need to be clear about what evidence they include in their reasoning, and how.”
This approach is a radical departure from the way traditional policy reports are written. In such reports, it is often difficult to understand what decisions were made when models were constructed, why certain inputs were used rather than others, or even which data were used. “Our analysis is fully transparent and reproducible, and we would love to see opposing arguments laid out in a similar way,” say Saez and Zucman. “To truly achieve the ideals of evidence-based policy, both policymakers and policy analysts need to be clear about what evidence they include in their reasoning, and how.”
Analysts representing different policy areas and political inclinations have already endorsed the OPA approach. The 2018 BITSS Annual Meeting, which gathered leaders in the research transparency movement, hosted a panel of policy analysts including Hilary Hoynes of UC Berkeley, Matt Jensen of the American Enterprise Institute, Kevin Perese of the US Congressional Budget Office, and Josh Rosenberg of GiveWell. Moderated by Hoces de la Guardia, the panel discussed various challenges to the rigor and credibility of policy analysis, and agreed that greater transparency and reproducibility may indeed transform the discipline.
To enable the use of OPA, BITSS has provided details on tools and best practices for improved transparency and reproducibility in policy analysis through a set of guidelines. And like any other good evidence-based organization, we are taking lessons to heart; a more detailed draft will be available in the coming weeks. BITSS invites feedback from anyone in the policy analysis community (questions and suggestions may be sent to Fernando Hoces de la Guardia at firstname.lastname@example.org).
This post was also shared on the CEGA blog.