In a standard scientific analysis, one analyst or team presents a single analysis of a data set. However, there are often a variety of defensible analytic strategies that could be used on the same data. Variation in those strategies could produce very different results.

In this project, we introduce the novel approach of “crowdsourcing a dataset.” We hope to recruit multiple independent analysts to investigate the same research question on the same data set in whatever manner they see as best. This approach should be especially useful for complex data sets in which a variety of analytic approaches could be used, and when dealing with controversial issues about which researchers and others have very different priors. If everyone comes up with the same results, then scientists can speak with one voice. If not, the subjectivity and conditionality on analysis strategy are made transparent.

This first project establishes a protocol for the independent simultaneous analysis of a single dataset by multiple teams and resolution of the variation in analytic strategies and effect estimates among them. The research question for this first attempt at crowdsourcing is as follows: Are soccer referees more likely to give red cards to dark skin toned players than light skin toned players?

Find the most recent version of Christensen’s analysis here. Learn more about the project here.