Research shows that women are less likely to point out and penalize mistakes in science, and publish fewer comments and failed replications in scientific journals. What does this mean for the social sciences?
BITSS Program Manager Grace Han interviews David Klinowski (Visiting Assistant Professor at the University of Pittsburgh Katz Graduate School of Business) about his forthcoming publication, “Voicing Disagreement in Science: Missing Women.”
What does it mean for the state of social science when half of all published studies fail to replicate? A recent Nature article reported that, according to some studies, 35–70 percent of published results don’t support replication with new data. “Often, researchers cannot even reproduce results when using the same data and code as the original paper, because key information is missing,” the authors wrote.
Despite glaring issues with the credibility of published research, replicating results remains disincentivized, and stark inequalities in their production and publication exacerbate this challenge. At the 2023 BITSS Annual Meeting, David Klinowski, visiting assistant professor at the University of Pittsburgh Katz Graduate School of Business, discussed determinants of gender gaps in education and labor markets. His forthcoming publication, Voicing Disagreement in Science: Missing Women, shows that women are less likely to point out and penalize mistakes in science and publish fewer comments and failed replications in scientific journals.
We talked to David about why women are missing from academic criticisms and what this means for the social sciences. Responses have been edited for length and clarity.
Grace Han: Your research indicates that women are missing from critiques and failed replications. Why do you think that is?
David Klinowski: Many factors may influence a researcher’s decision to criticize a published paper, some of which may have a gender component. For example, a researcher may be more or less willing to express criticism depending on her level of seniority or whether she has coauthors to pursue the criticism with. If women tend to be more junior than men, or have fewer coauthors, then these differences could contribute to the gender gap in critiques and failed replications.
Opportunity costs may also play a role. Our profession values critiques and replications much less than it values novelty. Researchers may naturally choose to focus on doing original work rather than attempting to replicate or criticize others’ work.
Women may also suffer backlash to a larger degree than men for criticizing work. Research has found that women are often penalized for displaying male-stereotypical behaviors, such as assertiveness. If writing a critique or a failed replication is perceived as confrontational, then women may be reluctant to write these papers for fear of backlash. It is also possible that women criticize as much as men do but their criticisms are more often rejected during peer review.
What are the impacts of women’s comparative absence from such criticisms?
DK: It is hard to say. Publishing a critique in a top journal could advance the critic’s career. She could gain recognition from her peers, and citations to her work. And the process of replicating someone’s work could be a great learning experience. But if negative consequences such as backlash are larger for women, then it may be entirely rational for women not to participate.
We should also worry about what impact the lack of female-authored critiques and failed replications has on science itself. Is the process of self-correction in science slower as a result? Are there views and insights that never emerge as a result?
What does the relative absence of women from critical publications say about the scientific literature and publishing process at large?
DK: One way to interpret the results is that they illustrate how difficult and fraught it is to express post-publication criticism. It is not always easy to find out whether a replication exists, or to know what a critique may imply about the original paper. Original authors may be terrified of scrutiny, and may not be eager to share their code, data, or other details of their work.
What can be done to increase women’s participation in academic criticisms? Is this wise given the potential risks and professional blowback?
DK: I think a promising approach is to make it easier and more rewarding for researchers to participate. To even start to scrutinize or replicate a paper, researchers must be able to access its data, code, and methods. BITSS does a great job at educating scholars on why and how to do transparent and reproducible research.
One way to prevent authors from feeling singled out when their work is criticized or replicated may be to conduct replications via a “replication audit,” an organized effort by multiple researchers to replicate a large number of studies. The Replication Games [convened by I4R,] is one such effort. Researchers may find it more attractive to participate in replication audits than to replicate a single study on their own.
Afterwards, replicators become coauthors of a meta-paper that combines the different replications conducted, but their names are not associated with the particular paper they replicated. To increase participation of women and other researchers in academic criticisms, we need more of this type of careful design of the incentives and institutions.
In March, alongside the American Economic Association’s Committee on the Status of Women in the Economics Profession and Economists for Equity at UC Berkeley, CEGA will co-host the Berkeley Replication Games, which will focus on the participation of underrepresented researchers.
To find out more, check out our event page.