BITSS’ Take on Star Wars: The Force Awakens

No, not really, but it does seem that the American Economic Association has fairly good timing with the publication of Brodeur, Lé, Sangnier, and Zylberberg’s “Star Wars: The Empirics Strikes Back” as the lead article in the new issue of American Economic Journal: Applied Economics, which appeared in my inbox this morning. We had a guest post by the lead author Abel Brodeur on the blog back in August.

From the abstract:

Using 50,000 tests published in the AER, JPE, and QJE, we identify a residual in the distribution of tests that cannot be explained solely by journals favoring rejection of the null hypothesis. We observe a two-humped camel shape with missing p-values between 0.25 and 0.10 that can be retrieved just after the 0.05 threshold and represent 10–20 percent of marginally rejected tests. Our interpretation is that researchers inflate the value of just-rejected tests by choosing “significant” specifications. We propose a method to measure this residual and describe how it varies by article and author characteristics.

And the tell-all Figure 1:

Brodeur Figure 1
Brodeur et al Figure 1

Basically, there’s a lot of p-hacking going on in economics. (I think 10-20% counts as a lot.) The complete paper is here, and an ungated earlier version is here.

Leave a Reply