By Alex Eble (Brown University), Peter Boone (Effective Intervention), and Diana Elbourne (University of London)
The randomized controlled trial (RCT) now has pride of place in much applied work in economics and other social sciences. Economists increasingly use the RCT as a primary method of investigation, and aid agencies such as the World Bank and the Gates Foundation may rely on the RCT to determine which projects to fund.
Motivating this use of the method is the notion that RCT results give us the “gold standard” of evidence in understanding causal relationships, something that studies using observational data fall short of. Unfortunately, this is the case only if certain conditions are satisfied in the design, conduct, and reporting of RCTs, and we find that in a large proportion of RCTs run by economists, they aren’t.
The RCT has been used extensively in medicine for the past 70 years, with hundreds of thousands of trials registered at clinicaltrials.gov, the US national trial registry, alone. In the last two decades, a group of scientists has investigated the impact of flaws in trial design, conduct and reporting on the treatment effect estimates generated by trials. They’ve found that a set of mistakes, which past trials have often stumbled into, lead directly to biased results. How biased? Treatment effects of trials with these mistakes are found to be 10-30% larger than trials that don’t make these mistakes, and in several cases, these mistakes have led to spurious conclusions of treatment efficacy.
The most prominent results of this work are the CONSORT Statement, a set of guidelines for trialists to follow in reporting of trials, and the Cochrane Handbook, a tool for scholars performing meta-analyses which helps them evaluate the quality of evidence a study provides. These documents describe the mistakes made, help trialists design studies which avoid them, and provide readers with guidelines on how to identify the mistakes in reports of RCTs.
These mistakes overlap with much of what has been discussed earlier on this blog – failure to register trials leads to reporting bias, and failure to register pre-analysis plans results in the same trial generating suspiciously large numbers of positive findings which may or may not be statistical artefact – as well as many things which haven’t been discussed. The issues can be grouped into the following six categories:
- Selection bias: does the trial achieve balance between treatment groups in key prognostic factors (both known and unknown) at randomization, or was the randomization corrupted/deterministic?
- Performance bias: do the intervention providers in the trial know what treatment the participants are getting, and if so, can this knowledge bias the care they give; similarly, do participants in the trial know what treatment they’re getting, and if so, can this knowledge affect their behavior or bias the data they provide in favor of/against the treatment?
- Detection bias: do the data collectors have reason to skew the data collected in favor of/against the treatment?
- Attrition bias: is the balance between treatment arms achieved at randomization maintained throughout the trial?
- Reporting bias: are we seeing the full results of the trial, or just a subset of the analyses performed which have been cherry-picked because they are statistically significant?
- Sample size bias: is the study adequately powered to conclusively answer the question it poses?
In our recent working paper, Risk and Evidence of Randomized Controlled Trials in Economics, we use this medical literature on bias from trial design, conduct and reporting flaws, to evaluate the risk of these six biases in RCTs published in the economics literature between 2001 and 2011. We find that economists, despite attesting to lean on medicine in their use of the RCT, seem not to have taken advantage of this large literature on how to avoid bias. The papers that we collected and evaluated often stumbled into the exact same potholes that plagued medical trialists in the past and we find, as a whole, trials in economics are at much greater risk of having exaggerated treatment effects and biased conclusions than is necessary.
In order to avoid these mistakes and get the most precise, unbiased results possible from these studies, we propose that economists create and enforce a set of standards for trial design, conduct and reporting which draws heavily on the resources in CONSORT and Cochrane, but are adapted to the specific circumstances of trials in economics. In order to have bite, adherence to these standards must be required for publication in journals. Similar requirements have led to a large increase in the quality of reporting of trials in the medical literature. These standards could also be used to appropriately weight results from previous studies by the quality of evidence provided.
Much of the work has been done for us – the lessons in the CONSORT Statement and Cochrane Handbook are the result of decades of experimentation, rigorous scrutiny, and broad consensus on how to avoid bias in trials. To the extent that we aren’t using these lessons to improve our own trials, we are leaving a $100 bill on the ground…and economists hate that.