Kicking off a new partnership with The Choice Lab at the 68° North Conference

Jennifer Sturdy–BITSS Program Advisor Science, my lad, is made up of mistakes, but they are mistakes which it is useful to make, because they lead little by little to the truth. It seems fitting that I read a book which references this Jules Verne quote just before our BITSS panel on research…

It Depends…(and not on the weather)

A read out on key questions from Day 2 of our Research Transparency and Reproducibility Workshop (RT2) in Berkeley, California. Well, Day 1 of RT2 was foggy – but Day 2 brought the Northern California rain. If you participated in Day 2, or visited our OSF page to follow along on your…

Questions from Our Latest Workshop

Garret Christensen–BITSS Project Scientist I’m in Barcelona, where I delivered a reproducibility workshop and am attending the International Meeting on Experimental and Behavioral Social Sciences (IMEBESS).  The materials from the workshop are available on Github, as always. I got several good questions from the workshop participants, some methodological, some software, and I…

Reflections from BITSS’s First Workshop in South Asia

By BITSS Program Manager Kelsey Mulcahy You’ve probably noticed the growing interest in research transparency and reproducibility issues and training – conversations with your colleagues, increasing numbers of high-profile panels – and of course, a number of BITSS workshops this Spring from UC Merced, California to Cuernavaca, Mexico to Delhi, India. We…

Links: Blind Analysis and Pre-Analysis Plans, Replication Failure

Garret Christensen–BITSS Project Scientist   There’s an interesting new proposal to deal with the problems of bias, p-hacking, and reproducibility failures from Saul Perlmutter, Nobel laureate UC Berkeley physicist and director of the Berkeley Institute for Data Science (where I’m a fellow) called blind analysis. Perlmutter and Robert MacCoun of Stanford have…

Emerging Researcher Perspectives: Replication as a Credible Pre-Analysis Plan

One of the most important tools for enhancing the credibility of research is the pre-analysis plan, or the PAP. Simply put, we feel more confident in someone’s inferences if we can verify that they weren’t data mining, engaging in motivated reasoning, or otherwise manipulating their results, knowingly or unknowingly. By publishing a…

Emerging Researcher Perspectives: Get it Right the First Time!

Guest post by Olivia D’Aoust, Ph.D. in Economics from Université libre de Bruxelles, and former Fulbright Visiting Ph.D. student at the University of California, Berkeley. As a Fulbright PhD student in development economics from Brussels, my experience this past year on the Berkeley campus has been eye opening. In particular, I discovered…

Three Transparency Working Papers You Need to Read

Garret Christensen, BITSS Project Scientist Several great working papers on transparency and replication in economics have been released in the last few months. Two of them are intended for a symposium in The Journal of Economic Perspectives, to which I am very much looking forward, and are about pre-analysis plans. The first of…

Call for Papers: Working Group in African Political Economy (WGAPE)

BITSS is co-sponsoring the 4th Working Group in African Political Economy (WGAPE) Annual Meeting, taking place May 29-30 at the Watson Institute for International and Public Affairs, Brown University. WGAPE brings together faculty and advanced graduate students in Economics and Political Science who combine field research experience in Africa with training in…

Creating Standards for Reproducible Research: Overview of COS Meeting

By Garret Christensen (BITSS) Representatives from BITSS (CEGA Faculty Director Ted Miguel, CEGA Executive Director Temina Madon, and BITSS Assistant Project Scientist Garret Christensen–that’s me) spent Monday and Tuesday of this week at a very interesting workshop at the Center for Open Science aimed at creating standards for promoting reproducible research in the social-behavioral…

First Swedish Graduate Student Training in Transparency in the Social Sciences

Guest Post by Anja Tolonen (University of Gothenburg, Sweden) Seventeen excited graduate students in Economics met at the University of Gothenburg, a Monday in September, to initiate an ongoing discussion about transparency practices in Economics. The students came from all over the world: from Kenya, Romania, Hong Kong, Australia and Sweden of course. The initiative…

New Study Sheds Light on File Drawer Problem

A new study recently published in Science provides striking insights into publication bias in the social sciences: Stanford political economist Neil Malhotra and two of his graduate students examined every study since 2002 that was funded by a competitive grants program called TESS (Time-sharing Experiments for the Social Sciences). TESS allows scientists…

Call for Pre-analysis Plans of Observational Studies

Observational Studies is a peer-reviewed journal that publishes papers on all aspects of observational studies. Researchers from all fields that make use of observational studies are encouraged to submit papers. Observational Studies encourages submission of study protocols (pre-analysis plans) for observational studies. Before examining the outcomes that will form the basis for…

Call for Papers on Research Transparency

BITSS will be holding its 3rd annual conference at UC Berkeley on December 11-12, 2014. The goal of the meeting is to bring together leaders from academia, scholarly publishing, and policy to strengthen the standards of openness and integrity across social science disciplines. This Call for Papers focuses on work that elaborates new tools and strategies to increase the transparency and reproducibility of research. A committee of…

The Controversy of Preregistration in Social Research

Guest post by Jamie Monogan (University of Georgia) A conversation is emerging in the social sciences over the merits of study registration and whether it should be the next step we take in raising research transparency. The notion of study registration is that, prior to observing outcome data, a researcher can publicly…

Deadline Extended for BITSS Summer Institute!

The deadline for applying to the BITSS Summer Institute (June 2-5, 2014 – Berkeley, CA) has been extended to Wednesday, April 2. The week-long course will cover a broad array of topics related to openness and transparency in social science research, including the theory and implementation of pre-analysis plans; building reproducible knowledge; emerging techniques to carry…

Deadline Coming Up for BITSS Summer Institute!

Applications for the first BITSS Summer Institute (June 2-5, 2014 – Berkeley, CA) are due by Wednesday, March 26. The workshop will inform participants about the latest trends in the shift towards increased transparency in the social sciences, providing an overview of new tools and techniques for rigorous research in economics, political science, psychology,…

Preregistration: Not just for the Empiro-zealots

Leif Nelson making the case for pre-registration: I recently joined a large group of academics in co-authoring a paper looking at how political science, economics, and psychology are working to increase transparency in scientific publications. Psychology is leading, by the way. Working on that paper (and the figure below) actually changed my…

AEA 2014: The Use of Pre-analysis Plans and Other Transparency Approaches in Economics

By Guillaume Kroll (CEGA) BITSS held a session on research transparency at the 2014 Annual Meeting of the American Economic Association on January 5 in Philadelphia. UC Berkeley economist Ted Miguel, who was presiding over the session, kicked off the discussion by highlighting the growing interest in transparency across social science disciplines.…

BITSS Affiliates Advocate for Higher Transparency Standards in Science Magazine

In the January 3, 2014 edition of Science Magazine, an interdisciplinary group of 19 BITSS affiliates reviews recent efforts to promote transparency in the social sciences and make the case for more stringent norms and practices to help boost the quality and credibility of research findings. The authors, led by UC Berkeley…

Trying out the new Trial Registries

Reblogged from World Bank’s David McKenzie: Both the American Economic Association and 3ie have launched Impact Evaluation Trial Registries […] I recently tried out both registries by registering a couple of studies I have underway, so thought I’d share some feedback on the process for those of you wondering whether/how to register. Read…

Bias Minimization Lessons from Medicine – How We Are Leaving a $100 Bill on the Ground

By Alex Eble (Brown University), Peter Boone (Effective Intervention), and Diana Elbourne (University of London) The randomized controlled trial (RCT) now has pride of place in much applied work in economics and other social sciences. Economists increasingly use the RCT as a primary method of investigation, and aid agencies such as the World…

AEA RCT Registry Webinar This Friday

The American Economic Association’s RCT Registry is a registration tool for pre-analysis plans of Randomized Controlled Trials in economics and other social sciences. The Abdul Latif Jameel Poverty Action Lab (J-PAL) will be hosting a brown bag webcast this Friday, September 20th at 1pm (EDT) to go over the motivations behind the registry and…

Transparency-Inducing Institutions and Legitimacy

By Kevin M. Esterling (Political Science, UC Riverside) Whenever I discuss the idea of hypothesis preregistration with colleagues in political science and in psychology, the reactions I get typically range from resistance to outright hostility. These colleagues obviously understand the limitations of research founded on false-positives and data over-fitting. They are even…

The Need for Pre-Analysis: First Things First

By Richard Sedlmayr (Philanthropic Advisor) When we picture a desperate student running endless tests on his dataset until some feeble point finally meets statistical reporting conventions, we are quick to dismiss the results. But the underlying issue is ubiquitous: it is hard to analyze data without getting caught in a hypothesis drift,…

Freedom! Pre-Analysis Plans and Complex Analysis

By Gabriel Lenz (UC Berkeley) Like many researchers, I worry constantly about whether findings are true or merely the result of a process variously called data mining, fishing, capitalizing on chance, or p-hacking. Since academics face extraordinary incentives to produce novel results, many suspect that “torturing the data until it speaks” is…

Transparency and Pre-Analysis Plans: Lessons from Public Health

By David Laitin (Political Science, Stanford) My claim in this blog entry is that political science will remain principally an observation-based discipline and that our core principles of establishing findings as significant should consequently be based upon best practices in observational research. This is not to deny that there is an expanding…

Targeted Learning from Data: Valid Statistical Inference Using Data Adaptive Methods

By Maya Petersen, Alan Hubbard, and Mark van der Laan (Public Health, UC Berkeley) Statistics provide a powerful tool for learning about the world, in part because they allow us to quantify uncertainty and control how often we falsely reject null hypotheses. Pre-specified study designs, including analysis plans, ensure that we understand…

Bayes’ Rule and the Paradox of Pre-Registration of RCTs

By Donald P. Green (Political Science, Columbia) Not long ago, I attended a talk at which the presenter described the results of a large, well-crafted experiment. His results indicated that the average treatment effect was close to zero, with a small standard error. Later in the talk, however, the speaker revealed that…

An Open Discussion on Promoting Transparency in Social Science Research

By Edward Miguel (Economics, UC Berkeley) This CEGA Blog Forum builds on a seminal research meeting held at the University of California, Berkeley on December 7, 2012. The goal was to bring together a select interdisciplinary group of scholars – from biostatistics, economics, political science and psychology – with a shared interest…