by Abel Brodeur Introduction from BITSS: Today on the BITSS blog, Catalyst Abel Brodeur presents the Institute for Replication (I4R) (Twitter: @I4Replication), a new initiative for improving the credibility of published research. Read on to learn about I4R’s goals, its connections to the Social Science Reproduction Platform, and how to get involved.…
Introducing the Social Science Reproduction Platform, a resource for teaching and improving computational reproducibility
By Aleksandar Bogdanoski (Project Manager, BITSS), Fernando Hoces de la Guardia (Project Scientist, BITSS), Edward Miguel (Faculty Director, CEGA/BITSS), and Lars Vilhuber (Executive Director, Labor Dynamics Institute, Cornell University, and Data Editor, American Economic Association). This post is also posted on the CEGA blog. Do you teach, or participate in, an empirical…
Internal replication: another tool for the reproducibility toolkit
By Jade Benjamin-Chung (University of California, Berkeley) and Benjamin F. Arnold (University of California, San Francisco) Introduction from BITSS: Internal replication is a new tool in the reproducibility toolkit with which original study investigators replicate findings prior to submission to a peer-reviewed journal. Jade Benjamin-Chung (UC Berkeley) and Benjamin Arnold (UCSF) describe…
Three reasons in favor of transparent, reproducible, and ethical research practices
By Fernando Hoces de la Guardia (BITSS) and Sebastián Martínez (Inter-American Development Bank) This post is cross-posted on the IDB Impact blog and the CEGA blog. You can read it in Spanish here. Introduction from BITSS: This post highlights the results of a successful partnership between BITSS and the Inter-American Development Bank…
ReproducibiliTea: A Reproducibility-themed Journal Club
ReproducibiliTea started as a journal club at the University of Oxford and is now also a podcast co-hosted by Sam Parsons, Amy Orben, and Sophia Crüwell. As the name suggests, the leaders are focused on research reproducibility and subtopics including transparency and rigor. BITSS Catalyst Amy Riegelman interviewed Sam Parsons who responded…
Interpretation of study results (Part 2/2): A reproducible method
Guest post by Arnaud Vaganay (Meta-Lab) This post is the second of two dedicated to the reproducible interpretation of empirical results in the social sciences. Read part 1 here. In my previous post on the interpretation of study results, I contrasted the notions of: Analytic reproducibility, which is concerned with the reproducibility…
Interpretation of study results (Part 1/2): One reproducibility crisis may hide another
Guest post by Arnaud Vaganay (Meta-Lab) This post is the first of two dedicated to the reproducible interpretation of empirical results in the social sciences. Read part 2 here. If you are a regular reader of this blog, chances are high that you know all about the ‘reproducibility crisis’ that has struck…
Some SSMART reflections on who is doing research on transparency and doing that research transparently!
BITSS is delighted with the steady release of the results from our SSMART-funded research portfolio, and with the incredible response to some of the projects. It goes without saying that SSMART aimed to fund research on topics that would help fuel improved transparency and reproducibility in the social sciences – this meant…
Questions from Our Latest Workshop
Garret Christensen–BITSS Project Scientist I’m in Barcelona, where I delivered a reproducibility workshop and am attending the International Meeting on Experimental and Behavioral Social Sciences (IMEBESS). The materials from the workshop are available on Github, as always. I got several good questions from the workshop participants, some methodological, some software, and I…
Reproducibility of Research: Issues and Proposed Remedies – A Sackler Colloquium Reflection
Guest post: Cynthia M. Kroeger, Postdoctoral Fellow, Nutrition Obesity Research Center, University of Alabama at Birmingham From March 8-10, 2017, I had the honor of joining scientific researchers, publishers, journalists, entrepreneurs, and funders from all over the world to discuss Reproducibility of Research: Issues and Proposed Remedies. This Sackler colloquium of the…
Our 2017 ASSA Session
Garret Christensen –BITSS Project Scientist BITSS organized a session on meta-analysis and reproducibility in economics for the recent Allied Social Sciences Associations (ASSA, but better known as the AEAs, the largest annual economics conference and job market for PhD economists) in Chicago. Chicago, as much of the country, was in the midst…
BITSS/IPA Kenya Workshop on data-sharing and reproducibility
BITSS and Innovations for Poverty Action (IPA) recently co-hosted a workshop on reproducibility outside Nairobi, Kenya. IPA’s Stephanie Wykstra wrote a post for the IPA blog about the event: During the workshop (agenda here) we covered topics on how to make research reproducible. The sessions included an overview of why research is unreliable and…
Project TIER Faculty Fellowship Opportunities
Our friends at Project TIER have a new funding opportunity for faculty interested in teaching reproducible empirical methods: Project TIER is pleased to announce new rounds of two programs for social science faculty interested in training undergraduate and graduate students in the social sciences in transparent and reproducible methods of research. TIER…
Upcoming Events & Recent Papers
Garret Christensen–BITSS Project Scientist How has registration affected publication bias in medical research? This comes as a surprise to me, but maybe not so much, says Grant M. Gordon. He’s presenting on this research Monday 1/25 in the BIDS space (190 Doe Library) at UC Berkeley. Find the paper and more…
Open Source Software for Reproducible Social Science
Garret Christensen –BITSS Project Scientist BITSS offers grad student workshops in reproducible research, where we give a hands-on introduction to software that can help make your work more reproducible. A lot of the software is listed on the Software section of our Resources page, but I wanted to create a quick…
BITSS Sessions Around the World
Garret Christensen–BITSS Project Scientist I’ve recently had the opportunity to represent BITSS at a few interesting meetings and conferences that you might be interested to hear about. A group of political scientists and other social scientists met at Stanford and held a daylong workshop to discuss steps the discipline could take to…
Workshops in Kenya on Research Transparency
Garret Christensen–BITSS Project Scientist BITSS has recently launched an initiative to expand research transparency efforts in developing countries. I think this is a very good fit for BITSS–we are housed within the Center for Effective Global Action (CEGA), our faculty lead Ted Miguel is a preeminent development economist, our program director…
TIER Faculty Fellowships 2015-16
Richard Ball, Associate Professor of Economics, and Norm Medeiros, Associate Librarian, of Haverford College, are co-principal investigators of Project TIER. They are seeking the first class of TIER Fellows to promote and extend teaching of transparent and reproducible empirical research methods. Project TIER (Teaching Integrity in Empirical Research) is an initiative that promotes…
Join an Open Call on Reproducibility Tomorrow at 11 am (ET)
BITSS Project Scientist Garret Christensen will be participating in a discussion with the Mozilla Science Lab on reproducibility in research tomorrow at 11 am ET. The call is open to the public. For those interested in joining, more information can be found here.
This Monday at AEA2015: Transparency and Integrity in Economic Research Panel
This January 5th, 10.15am at the American Economic Association Annual Meeting in Boston, MA (Sheraton Hotel, Commonwealth Room). Session: Promoting New Norms for Transparency and Integrity in Economic Research Presiding: Edward Miguel (UC Berkeley) Panelists: Brian Nosek (University of Virginia): “Scientific Utopia: Improving Openness and Reproducibility in Scientific Research” Richard Ball (Haverford College): “Replicability…
Scientists Have a Sharing Problem
Dec 15th Maggie Puniewska posted an article in the Atlantic Magazine summarizing the obstacles preventing researchers from sharing their data. The article asks if “science has traditionally been a field that prizes collaboration […] then why [are] so many scientists stingy with their information.” Puniewska outlines the most cited reasons scientists reframe…
Scientific Irreproducibility and the Prospects of Meta-Research
A recent article from The Economist featuring John Ioannidis’ Meta-Research Innovation Center (METRICS), whose work to advance the credibility of research will be presented next week at the BITSS Annual Meeting. “Why most published research findings are false” is not, as the title of an academic paper, likely to win friends in the ivory tower. But it has certainly…
Scientific consensus has gotten a bad reputation—and it doesn’t deserve it
In a recent post, Senior science editor at Ars Technica, John Timmer defends the importance of consensus. Opening with the following quote from author Michael Crichton: Let’s be clear: the work of science has nothing whatever to do with consensus. Consensus is the business of politics. Science, on the contrary, requires only one investigator…
Teaching Integrity in Empirical Research
Richard Ball (Economics Professor at Haverford College and presenter at the 2014 BITSS Summer Institute) and Norm Medeiros (Associate Librarian at Haverford College) in a recent interview appearing on the Library of Congress based blog The Signal, discussed Project TIER (Teaching Integrity in Empirical Research) and their experience educating students how to…
Reproducible Research: True or False?
Keynote speaker at the upcoming BITSS annual meeting John Ioannidis (Professor of Health Research and Policy at Stanford School of Medicine, and Co-Director of the Meta-Research Innovation Center) speaks at Google about its efforts to improve research designs standards and reproducibility in science. Ioannidis is the author of the 2005 highly influential paper Why Most Published Research Findings Are False,…
Can Greater Transparency Lead to Better Social Science?
In a recent article on the Monkey Cage, professors Mike Findley, Nathan Jensen, Edmund Malesky and Tom Pepinsky discuss publication bias, the “file drawer problem” and how a special issue of the journal Comparative Political Studies will help address these problems. Similar to a recent article by Brendan Nyhan, reposted on the BITSS blog, the university professors writing…
MCC’s First Open Data Challenge
The U.S. Government’s Millennium Challenge Corporation (MCC) wants to hear your new and innovative ideas on how to maximize the use of data that MCC finances for its independent evaluations. Keynote speakers at this year’s BITSS Research Transparency Forum, Jennifer Sturdy and Jack Molyneaux at MCC’s Department of Policy and Evaluation, and Kathy Farley…
MCC's First Open Data Challenge
The U.S. Government’s Millennium Challenge Corporation (MCC) wants to hear your new and innovative ideas on how to maximize the use of data that MCC finances for its independent evaluations. Keynote speakers at this year’s BITSS Research Transparency Forum, Jennifer Sturdy and Jack Molyneaux at MCC’s Department of Policy and Evaluation, and Kathy Farley…
White House Calls for Comments on Reproducible Research
The White House’s Office of Science and Technology Policy (OSTP) has released a request for information on improving the reproducibility of federally funded scientific research. Given recent evidence of the irreproducibility of a surprising number of published scientific findings, how can the Federal Government leverage its role as a significant funder of scientific research…
Political Scientists Launch New Replication Initiative
Following a groundswell of interest for replications in the political sciences, first noticed from survey results posted on the Monkey Cage Blog, Political Scientists Seth Werfel (Stanford University) and Nicole Janz (Cambridge University), and research consultant Stephanie Wykstra launched the Political Science Replication Initiative, a new repository for uploading study replications. Increasingly, methodological political scientists have recognized…
eLife, the Center for Open Science, and Science Exchange partner to assess the reproducibility of cancer biology research
eLife will be the publisher for the results of the Reproducibility Project: Cancer Biology, an effort led by the Center for Open Science and Science Exchange. First announced in October 2013, with $1.3 million in funding from the Laura and John Arnold Foundation, The Reproducibility Project: Cancer Biology aims to replicate key…
Call for Papers on Research Transparency
BITSS will be holding its 3rd annual conference at UC Berkeley on December 11-12, 2014. The goal of the meeting is to bring together leaders from academia, scholarly publishing, and policy to strengthen the standards of openness and integrity across social science disciplines. This Call for Papers focuses on work that elaborates new tools and strategies to increase the transparency and reproducibility of research. A committee of…
Data Science Meets Social Science (Video)
The video from a recent BITSS roundtable entitled “Data Science Meets Social Science” is now available online. Organized in partnership with the UC Berkeley D-Lab, the event brought together leading social scientists and Silicon Valley professionals to discuss pathways of collaboration between the two different fields, and their increasing impact on society in the…
Significance Chasing in Research Practice
A new paper by Jennifer Ware and Marcus Munafò (University of Bristol, UK) Background and Aims The low reproducibility of findings within the scientific literature is a growing concern. This may be due to many findings being false positives which, in turn, can misdirect research effort and waste money. Methods We review factors that…
Replication in Economics Database
For scientific progress, it is pivotal to review research findings by independently replicating results, thus making the findings more reliable. However, in econometric research, it is not yet common practice to publish replication findings. Replication Wiki This wiki, developed by researchers at the University of Göttingen (Germany), compiles replications of empirical studies in economics.…
Tonight! Data Science Meets Social Science
Tonight in Berkeley! June 5, 2014 6:00pm — 7:30pm. David Brower Center (Kinzie Room), 2150 Allston Way. All across the social sciences we can see a convergence around the ideals of openness and reproducibility. Over the past years, the injection of ways of thinking and working from scientific computing into social science research has helped…
Reproducibility Session @ AAAS Forum on Science and Technology Policy
A core value of science is the ability to reproduce the findings of others in order to check for methodological rigor, errors, plausible interpretations, and/or misconduct. However, for a variety of reasons, from lack of time or resources to little professional recognition or credit for doing so, science may be failing to…
New book: Implementing Reproducible Research
New book from Victoria Stodden, Friedrich Leisch, and Roger D. Peng: “Implementing Reproducible Research“. In many of today’s research fields, including biomedicine, computational tools are increasingly being used so that the results can be reproduced. Researchers are now encouraged to incorporate software, data, and code in their academic papers so that others can…
Reproducibility Pioneers: Political Scientists Assigning Replications to Students
Deadline Extended for BITSS Summer Institute!
The deadline for applying to the BITSS Summer Institute (June 2-5, 2014 – Berkeley, CA) has been extended to Wednesday, April 2. The week-long course will cover a broad array of topics related to openness and transparency in social science research, including the theory and implementation of pre-analysis plans; building reproducible knowledge; emerging techniques to carry…
Replication Panels at ISA 2014
The 55th Annual Convention of the International Studies Association (March 26-29, 2014 – Toronto, Canada) will feature two panels on replication and reproducibility: Friday, March 28 8:15 AM – 10:00 AM Replication in International Relations: How Journal Data Policies and Replication in Teaching Can Improve Reproducibility Standards Friday, March 28 10:30 AM…
Deadline Coming Up for BITSS Summer Institute!
Applications for the first BITSS Summer Institute (June 2-5, 2014 – Berkeley, CA) are due by Wednesday, March 26. The workshop will inform participants about the latest trends in the shift towards increased transparency in the social sciences, providing an overview of new tools and techniques for rigorous research in economics, political science, psychology,…
Git can facilitate greater reproducibility and increased transparency in science
A well-articulated piece on why Git is a useful too for open, reproducible science (by Karthik Ram): Reproducibility is the hallmark of good science. Maintaining a high degree of transparency in scientific reporting is essential not just for gaining trust and credibility within the scientific community but also for facilitating the development of new ideas.…
"Creating tools is not the biggest challenge. The biggest challenge is getting people to use them"
Brian Nosek, Perspective from Psychology This is the third post of a video series in which we ask leading social science academics and experts to discuss research transparency in their discipline. The interview was recorded on December 13, 2013 at the University of California, Berkeley.
“Creating tools is not the biggest challenge. The biggest challenge is getting people to use them”
Brian Nosek, Perspective from Psychology This is the third post of a video series in which we ask leading social science academics and experts to discuss research transparency in their discipline. The interview was recorded on December 13, 2013 at the University of California, Berkeley.
Scientific Pride and Prejudice
Or why Jane Austen might well be the first game theorist. Science is in crisis, just when we need it most […] A major root of the crisis is selective use of data. Scientists, eager to make striking new claims, focus only on evidence that supports their preconceptions. Psychologists call this “confirmation bias”:…
Science Magazine Embarks on Reproducibility Quest
Marcia McNutt, Editor-in-Chief of Science, in the January 17 editorial: Science advances on a foundation of trusted discoveries. Reproducing an experiment is one important approach that scientists use to gain confidence in their conclusions. Recently, the scientific community was shaken by reports that a troubling proportion of peer-reviewed preclinical studies are not…
Replication & Reproducibility 2013: The best stories
Replication & Reproducibility 2013: The best stories
Git/GitHub, Transparency, and Legitimacy in Quantitative Research
Reblogged from The Political Methodologist. A complete research project hosted on GitHub is reproducible and transparent by default in a more comprehensive manner than a typical journal mandated replication archive […] Maintaining your research project on GitHub confers advantages beyond the social desireability of the practice and the the technical benefits of using…