When is an error not an error?

Guest post by Annette N. Brown and Benjamin D. K. Wood on the World Bank Development Impact blog:

We are seeing a similar propensity for replication researchers to use the word “error” (or “mistake” or “wrong”) and for this language to cause contentious discussions between the original authors and replication researchers. The lesson we are learning is: To err is human, but to use the word “error” in a replication study is usually not divine […] The problem is that pure replication of an original author’s empirical analysis is often really complicated, not to mention time consuming. And what we’re seeing is that even relatively successful pure replications end up with many estimates that are just not quite the same as in the original article. Replication researchers are often quick to call these “errors”. But if two people conduct the same analysis on the same data, and they each get similar but not identical estimates, who is to say what is right and what is wrong?

Read more…