Searching 30,000 Psychology Articles for Statistical Errors

I recently came across this interesting article in psychology, which scanned over 30,000 articles in psychology for statistical reporting errors–checking test statistics against p-values, as some articles claim significance levels that don’t match up with their test statistics. The lead author, Michèle B. Nuijten, blogged about the article for Retraction Watch, where she wrote:

To reliably investigate these claims [of errors in the psychology literature], we wanted to study reporting inconsistencies at a large scale. However, extracting statistical results from papers and recalculating the p-values is not only very tedious, it also takes a LOT of time.

So we created a program known as “statcheck” to do the checking for us, by automatically extracting statistics from papers and recalculating p-values. Unfortunately, we recently found that our suspicions were correct: Half of the papers in psychology contain at least one statistical reporting inconsistency, and one in eight papers contain an inconsistency that might have affected the statistical conclusion.

Yikes. That sounds like a lot.

My mind immediately went to applying this in economics. Statcheck can currently only check results reported in text in APA style, and economics unfortunately has no such standard for in-text reporting. We typically report regression coefficients and standard errors, and then indicate significance levels with symbols, most often *, **, and ***. I imagine readers are dividing the coefficient by the standard error in their heads and thinking “sure, that’s greater than two.” I assume there are errors of this type in the economics literature, but I think other types of errors are probably more likely to alter the main conclusion of an article, such as miscoding a variable, and that wouldn’t be caught by an automated algorithm like this. Of course, to the extent there are errors in economics tables or statistics reporting, my question is, why on earth are authors not using outreg or estout to automate your tables? (Or stargazer or xtable if you use R.) Spit the table out to a LaTeX file, input that file in your LaTeX paper, and boom you’re done, with no statistical errors (at least of this variety). A step beyond that would be to automate your entire table using dynamic documents, which can be done in MarkDoc in Stata and R Markdown/knitr in R.

Leave a Reply