By Kevin M. Esterling (Political Science, UC Riverside)
Whenever I discuss the idea of hypothesis preregistration with colleagues in political science and in psychology, the reactions I get typically range from resistance to outright hostility. These colleagues obviously understand the limitations of research founded on false-positives and data over-fitting. They are even more concerned, however, that instituting a preregistry would create norms that would privilege prospective, deductive research over exploratory inductive and descriptive research. For example, such norms might lead researchers to neglect problems or complications in their data so as to retain the ability to state their study “conformed” to their original registered design.
If a study registry were to become widely used in the discipline, however, it would be much better if it were embraced and seen as constructive and legitimate. One way I think we can do this is by shifting the focus away from monitoring our colleagues’ compliance with registration norms, which implicitly privileges prospective research, and instead towards creating institutions that promote transparency in all styles of research, with preregistration being just one element of the new institutions for transparency.
Transparency solves the same problems that preregistration is intended to address, in that transparency helps other researchers to understand the provenance of results and enables researchers to value contributions for what they are. If scholars genuinely share the belief that data driven research has scientific merit, then there really should be no stigma for indicating that is how one reached one’s conclusions. Indeed, creating transparency should enable principled inductive research since it creates legitimacy for this research and it removes the awkward need to state inductive research as if it had been deductive.
Participating in transparency-inducing institutions not only places one’s research on firmer scientific footing (whether that science is inductive or deductive), but creates other benefits for individual researchers as well, since being transparent with one’s methods also allows earlier feedback on research designs and also will likely induce researchers to allocate more time on the front end of their studies when possible. And transparency norms should largely be self-enforcing. As Brian Nosek argued at the December meeting, as more of the discipline becomes transparent then it becomes harder and harder for others not to be transparent.
One useful way forward would be for the social science disciplines to promote Brian Nosek’s Open Science Framework (OSF). This framework gives researchers tools to collaborate, archive and version their work product. In addition, OSF enables researchers to make portions of their study public, such as lab notebooks, computer code, and dataset. And OSF provides a tool by which participants can preregister their studies using pre-defined registration templates. Each social science discipline could create sanctioned registration templates within OSF in order to promote registration as a tool to enhance transparency. Designing registration templates should be an inclusive process to ensure that all members of a discipline view the templates as legitimate and well-tailored to their research tasks. Within a discipline, many different constituencies could design frameworks that conform to different styles of research. For example, in political science, each organized section of APSA could develop one or more templates that accommodate the kind of research done in the section.
In political science, we have discussed the possibility of having the Society for Political Methodology (POLMETH) create a searchable/sortable database on its website that is a listing of registrations that have been verified as compliant. POLMETH can recommend that these verified registrations be in OSF but the registration could be elsewhere as well. By linking its official preregistration list with OSF, political science can promote registration norms, but within a framework that first and foremost promotes research transparency.
While our colleagues can cite a long list of arguments against preregistration, it’s difficult to think of principled arguments against transparency, and social scientists should find it easy to embrace transparency norms.
 |
|