An illustration of how the policy community can use OPA to strengthen the evidence-to-policy link. By Fernando Hoces de la Guardia, BITSS Project Scientist What is the deworming OPA? Opacity in policy analysis presents barriers to applying rigorously generated evidence to novel settings. Open Policy Analysis (OPA) is a framework to improve…
Welcome to the BITSS Blog
Reflections on the 9th BITSS Annual Meeting: Toward a More Open and Inclusive Scientific Ecosystem
By Katie Hoeberling and Aleksandar Bogdanoski We look forward each year to the BITSS Annual Meeting as a space for discussing new meta-research and open science initiatives, as well as for reuniting with old colleagues and welcoming new ones, normally with handshakes and hugs. This year has been anything but normal, though.…
Peer Community In Meta-Research: Community peer review
By Chris Hartgerink (Liberate Science, GmbH) Introduction from BITSS: BITSS launched MetaArXiv (formerly BITSS Preprints) in 2017 as a preprint service where researchers could share and discover work focused on improving research transparency and reproducibility and other meta-scientific research. We recently teamed up with the Peer Community in (PCI) Meta-Research to give MetaArXiv…
Internal replication: another tool for the reproducibility toolkit
By Jade Benjamin-Chung (University of California, Berkeley) and Benjamin F. Arnold (University of California, San Francisco) Introduction from BITSS: Internal replication is a new tool in the reproducibility toolkit with which original study investigators replicate findings prior to submission to a peer-reviewed journal. Jade Benjamin-Chung (UC Berkeley) and Benjamin Arnold (UCSF) describe…
Announcing the launch of the Social Science Prediction Platform!
By Aleksandar Bogdanoski (BITSS Senior Program Associate) and Katie Hoeberling (BITSS Program Manager). This post is also published on the Development Impact blog and the CEGA blog. What will be the effect of raising the minimum wage by $2 an hour? How will cash transfers impact local prices in rural Kenya? Will…
Why COVID-19 makes research transparency more important than ever
By Aleksandar Bogdanoski, Katie Hoeberling, and Fernando Hoces de la Guardia This post is also published on the CEGA blog. The COVID-19 pandemic has arguably had a more profound impact on scientific research than any other phenomenon in recent history; between January and May this year, over 23,000 research papers were written…
Open Scholarship Week 2020: Embracing openness in Ireland and beyond
by Hardy Schwamm (NUI Galway) and Elaine Toomey (University of Limerick/Cochrane Ireland) Introduction from BITSS: In what is becoming an annual event, the Open Science Community Galway at the National University of Ireland Galway (NUI Galway) hosted Open Scholarship Week 2020, featuring a series of workshops, seminars, and presentations dedicated to transparent,…
The 2019 BITSS Annual Meeting: A barometer for the evolving open science movement
By Aleksandar Bogdanoski and Katie Hoeberling Each year we look forward to our Annual Meeting as a space for showcasing new meta-research and discussing progress in the movement for research transparency. These meetings offer snapshots of the evolving priorities and challenges faced by the scientific community working to change research norms. BITSS’s…
New resources for de-identifying and publishing research data from J-PAL
Introduction from BITSS: Open data has many benefits. It can foster collaboration, facilitate more complete meta-analysis, and improve the visibility of related research outputs. At the same time, we know that re-identification can cause real risks to study participants and that balancing openness with such risks is a delicate and often difficult…
What development economists talk about when they talk about reproducibility…
Development economists, by and large, are driven to inform effective policy by generating rigorous and credible evidence. So in the face of failures to replicate findings, as well as evidence of pervasive publication bias across the social sciences, it’s no surprise that work to increase research transparency and reproducibility is underway in…
Ted Miguel in conversation on “Transparent and Reproducible Social Science Research: How to Do Open Science”
By Isabelle Cohen and Hagit Caspi A note from BITSS: Recently, we announced the official release of a new textbook, “Transparent and Reproducible Social Science Research: How to do open science,” by Garret Christensen, Jeremy Freese, and BITSS Faculty Director Ted Miguel. As the first textbook with a comprehensive overview of the…
Improving research transparency through easier, faster access to studies in the AEA RCT Registry
By Keesler Welch, Senior Research Associate (Trial Registry), J-PAL Global, and James Turitto, Senior Research Manager, J-PAL Global. This post was originally published here on the J-PAL Blog. J-PAL and the American Economic Association (AEA) have been working together over the past year to create digital object identifiers, or DOIs, for each…
Transparent and Reproducible Social Science Research: A new open science textbook
It’s been nearly 15 years since John Ioannidis’s “Why Most Published Research Findings are False” was published, turning the scientific community on its head. Today you might think little had changed. Just type “science is broken” into your online search engine, and you’ll find dozens of recent articles, blog posts, videos, and…
Pre-results Review at the Journal of Development Economics: Lessons learned so far
By Andrew Foster (Brown University), Dean Karlan (Northwestern University), Edward Miguel (UC Berkeley) and Aleksandar Bogdanoski (BITSS) This post was originally posted on the Development Impact blog. BITSS has been working with the Journal of Development Economics (JDE) to introduce Pre-results Review (also referred to as “registered reports” in other disciplines) as…
Pre-results review reaches the (economic) lab: Experimental Economics follows the Journal of Development Economics in piloting pre-results review
In its April 2019 issue, the journal Experimental Economics issued a Call for Submissions for a virtual Symposium of 5-7 papers to be published under “pre-results review”. BITSS Senior Program Associate Aleksandar Bogdanoski talked to Irenaeus Wolff of University of Konstanz, who along with Urs Fischbacher is a guest editor for the…
Better pre-analysis plans through design declaration and diagnosis
DeclareDesign is a set of software packages for planning for and assessing research designs, written by Graeme Blair, Jasper Cooper, Alexander Coppock, and Macartan Humphreys. The software is based on the Model-Inquiry-Data Strategy-Answer Strategy (MIDA) framework for declaring and diagnosing research designs introduced in their paper forthcoming at the American Political Science Review. In…
Opening up the analysis behind Elizabeth Warren’s wealth tax plan
Press release: BERKELEY, CA (Wednesday, March 13, 2019) — BITSS collaborated with UC Berkeley economists Emmanuel Saez and Gabriel Zucman to produce a fully reproducible version of their policy report for Sen. Elizabeth Warren’s wealth tax plan.
Three reasons in favor of transparent, reproducible, and ethical research practices
By Fernando Hoces de la Guardia (BITSS) and Sebastián Martínez (Inter-American Development Bank) This post is cross-posted on the IDB Impact blog and the CEGA blog. You can read it in Spanish here. Introduction from BITSS: This post highlights the results of a successful partnership between BITSS and the Inter-American Development Bank…
The Persistence of False Paradigms in Low-Power Sciences
By Pascal Michaillat*, Brown University It is commonly believed that the lack of experimental evidence typical in the social sciences slows but does not prevent the replacement of existing theories by newer, better ones. A simple model of scientific research and promotion challenges that belief, however. In the model, scientists are slightly…
The Future of Forecasting – Highlights from the BITSS Workshop on Forecasting Social Science Research Results
By Nicholas Otis, second-year PhD student in Health Economics at UC Berkeley This post is also published on the CEGA blog. Researchers are increasingly collecting forecasts of social science research results. For example, researchers have recently integrated predictions into studies examining questions like what are the long-term effects of a community-driven development…
Working toward a Common Rule for Transparent, Reproducible, and Ethical Research
by Jennifer Sturdy, BITSS Communalism, organized skepticism, disinterestedness, and universalism[1]. These scientific norms guide BITSS as an advocate for research transparency. From our perspective, research transparency is the means by which the research ecosystem operationalizes these scientific norms; and fully transparent research demonstrates when these norms are adhered to, and when they…
ReproducibiliTea: A Reproducibility-themed Journal Club
ReproducibiliTea started as a journal club at the University of Oxford and is now also a podcast co-hosted by Sam Parsons, Amy Orben, and Sophia Crüwell. As the name suggests, the leaders are focused on research reproducibility and subtopics including transparency and rigor. BITSS Catalyst Amy Riegelman interviewed Sam Parsons who responded…
Power to the Plan
By Clare Leaver, Owen Ozier, Pieter Serneels, and Andrew Zeitlin This post is also posted on the Development Impact blog. The holidays are upon us. You might like to show off a bit by preparing something special for the ones you love. Why not make a pre-analysis plan this holiday season? You’re…
Open Access: The timely publishing solution truly serving scientists, science and the public
This post, written by Elizabeth Marincola, Senior Advisor for Research Communications and Advocacy at the African Academy of Sciences (AAS), was originally published on the AAS Open Research blog. It has also been cross-posted on the CEGA blog. Note from Katie Hoeberling and the BITSS team: At BITSS, we actively work with…
Pre-results review at the Journal of Development Economics: Taking transparency in the discipline to the next level
This post, developed by Aleksandar Bogdanoski (Program Associate, BITSS) and Keesler Welch (Research Associate, J-PAL) with support from Anja Sautmann (Director of Research, Education, and Training, J-PAL), is also posted on the CEGA blog and J-PAL blog. Earlier this year, the Journal of Development Economics (JDE) began the pilot of a “pre-results…
A Great Day for Open Policy Analysis
Post by Fernando Hoces de la Guardia (BITSS Postdoc). How the best tweet of the year came not from Taylor Swift, or Barack Obama… but from CBO. Earlier this year, we wrote a blog post outlining our vision for bringing “Open Science” into policy analysis. The basic idea is simple: government policy analysts…
CONSORT-SPI 2018: Announcing an extension for randomized controlled trials of social and psychological interventions
Re-post by Paul Montgomery, Evan Mayo-Wilson, and Sean Grant Complete and transparent reporting of randomized controlled trials is integral for replication, critical appraisal and understanding context. Published today in Trials, a new extension of the CONSORT Statement aims to improve the reporting of randomized controlled trials of social and psychological interventions. Here,…
MetaLab Awards Three Contribution Challenge Prizes!
Guest announcement by Christina Bergmann and Sho Tsuji (MetaLab) The MetaLab challenge calling for meta-analyses on cognitive development, with support from Berkeley Initiative for Transparency in the Social Sciences (BITSS), has closed. We received data for 7 meta-analyses, which will be added to MetaLab in the coming months. The winners are three…
Research transparency in Sub-Saharan Africa: Lessons learned and ways forward
Guest post by Soazic Elise Wang Sonne (World Bank) Despite efforts by African governments to significantly raise public spending on scientific research, the continent, which is home to 14% of the world’s population, contributes to less than 1% of published research outputs (David Dunne, 2017). While this can be partly attributed to…
Interpretation of study results (Part 2/2): A reproducible method
Guest post by Arnaud Vaganay (Meta-Lab) This post is the second of two dedicated to the reproducible interpretation of empirical results in the social sciences. Read part 1 here. In my previous post on the interpretation of study results, I contrasted the notions of: Analytic reproducibility, which is concerned with the reproducibility…
Interpretation of study results (Part 1/2): One reproducibility crisis may hide another
Guest post by Arnaud Vaganay (Meta-Lab) This post is the first of two dedicated to the reproducible interpretation of empirical results in the social sciences. Read part 2 here. If you are a regular reader of this blog, chances are high that you know all about the ‘reproducibility crisis’ that has struck…
Transparency and Trust in the Research Ecosystem
CEGA launched the Berkeley Initiative for Transparency in the Social Sciences (BITSS) based on the argument that more transparency in research could address underlying factors driving publication bias, unreliable research findings, a lack of reproducibility in the published literature, and a problematic incentive structure within the research ecosystem. Meanwhile an ever-increasing number…
Open Science Comes To Policy Analysis
The Royal Society’s motto, “Take nobody’s word for it,” reflects a key principle of scientific inquiry: as researchers, we aspire to discuss ideas in the open, to examine our analyses critically, to learn from our mistakes, and to constantly improve. This type of thinking shouldn’t guide only the creation of rigorous evidence…
Registered Reports: Piloting a Pre-Results Review Process at the Journal of Development Economics
The world is a messy place. What happens when the results of an empirical study are mushy or inconsistent with prevailing theories? Unfortunately, papers with unclear or null results often go unpublished, even if they have rigorous research designs and good data. In such cases, the research community is typically only left…
Open Science and a Culture of Health: You Two Should Talk
Guest post by Sean Grant and Kathryn Bouskill (RAND Corporation) The Open Science movement aims to advance scientific progress by making research more open and accessible. The Culture of Health movement aims to advance collective well-being by making health a shared value and priority among all policy sectors, not just healthcare. Proponents…
2017 Annual Meeting Showcases Innovations for Transparent and Reproducible Science
By: Kelsey Mulcahy, BITSS Program Manager The 2017 Annual Meeting of the Berkeley Initiative for Transparency in the Social Sciences (BITSS) moved beyond the credibility crisis in social science research to discuss potential solutions. The two-day event featured a workshop, two panel discussions, an award ceremony for the 2017 Leamer-Rosenthal Prize Recipients,…
Navigating the Garden of Forking Paths
Guest post by Kweku Opoku-Agyemang (Center for Effective Global Action, Cornell Tech, and International Growth Centre) Improving ethical and transparency standards within and across the social and behavioral science professions appears to be more of a journey than a destination. Thankfully, this is a voyage that has an ever-growing number of travelers.…
Reflections from three years of Leamer-Rosenthal… and considerations for the future of open social science!
“I didn’t do my work for money or prizes – but for the excitement of discovery” – Edmund Phelps (Economist) BITSS recently announced the 2017 Leamer-Rosenthal Prize Recipients. The eight recipients, as well as the larger pool of nominees, are part of a growing movement in social science to practice better science,…
Diary of a Research Transparency Postdoc: South America Tour
Fernando Hoces de la Guardia – BITSS Postdoctoral Scholar I joined BITSS in September as its first postdoc and was excited to hit the ground running with a series of events to promote research transparency in Latin America. Through partnerships with local universities, we managed to organize back-to-back seminars in Chile, Peru…
The Power of Bias in Economics Research
Tom Stanley, friend of BITSS and one of the leaders of the Meta-Analysis of Economics Research Network (MAER-Net), recently published a paper in The Economic Journal with John Ioannidis and Chris Doucouliagos on statistical power in economics research that should be of interest to the BITSS network. From Tom: “The power of…
Recipients of 2017 Leamer-Rosenthal Prizes for Open Social Science
The Berkeley Initiative for Transparency in the Social Sciences (BITSS) awards eight prizes to researchers and educators demonstrating values and practices of openness and transparency in research. BERKELEY, CA (Thursday, October 12, 2017) – The open science community has grown rapidly in recent years, partly in response to a series of newsworthy…
Public Data that Isn’t (or Wasn’t) Public
–Guest Post: Justin Gallagher, Assistant Professor of Economics, Case Western Reserve University I recently completed a coauthored working paper, together with Paul J. Fisher, that examines whether electronic monitoring via red light traffic cameras leads to fewer vehicle accidents and injuries. As part of the project, we…
Review of Stata’s dyndoc
Guest post: Tomas Dvorak is a Professor of Economics at Union College and a former Project TIER fellow. This is a repost. The original post can be found here. As a huge fan of Stata I was super-excited about dynamic markdown documents newly available in the latest Stata 15 release. I played with…
Kicking off a new partnership with The Choice Lab at the 68° North Conference
Jennifer Sturdy–BITSS Program Advisor Science, my lad, is made up of mistakes, but they are mistakes which it is useful to make, because they lead little by little to the truth. It seems fitting that I read a book which references this Jules Verne quote just before our BITSS panel on research…
Some SSMART reflections on who is doing research on transparency and doing that research transparently!
BITSS is delighted with the steady release of the results from our SSMART-funded research portfolio, and with the incredible response to some of the projects. It goes without saying that SSMART aimed to fund research on topics that would help fuel improved transparency and reproducibility in the social sciences – this meant…
It Depends…(and not on the weather)
A read out on key questions from Day 2 of our Research Transparency and Reproducibility Workshop (RT2) in Berkeley, California. Well, Day 1 of RT2 was foggy – but Day 2 brought the Northern California rain. If you participated in Day 2, or visited our OSF page to follow along on your…
Trying to clear the fog
We had a lovely, foggy morning start to our Research Transparency and Reproducibility Training (RT2) here in Berkeley yesterday, and we have already had the opportunity to enjoy some great conversations with participants and faculty alike. One early suggestion brought to us by Arnaud Vaganay was the need to first frame what…
Announcing Open Enrollment for the BITSS MOOC: Transparent and Open Social Science Research
Most of you probably noticed that 2015 marked a turning point for BITSS. It was the year we launched our Social Science Meta-Analysis and Research Transparency (SSMART) Grant Program, the Leamer-Rosenthal Prizes for Open Social Science, and our new website. It was also the year we started focusing on two main objectives…
Opening the ‘black box’: the need for fidelity and process evaluations of behaviour change interventions
Guest Post: Elaine Toomey, Health Behaviour Change Research Group, National University of Ireland, Galway Sometimes the journey is just as, if not more, important than the destination. A motivational quote that perhaps graces one too many t-shirts of travellers seeking to ‘find themselves’, it is never more appropriate than when it comes…
Questions from Our Latest Workshop
Garret Christensen–BITSS Project Scientist I’m in Barcelona, where I delivered a reproducibility workshop and am attending the International Meeting on Experimental and Behavioral Social Sciences (IMEBESS). The materials from the workshop are available on Github, as always. I got several good questions from the workshop participants, some methodological, some software, and I…