As new technologies change how we engage with research, how can we speed up evidence synthesis without sacrificing quality? In this post, the BITSS team shares its approach through DEVIDENCE, a multi-institutional project building a transparent repository of development evidence.
AI is transforming research, including the way findings are interpreted and applied. In low- and middle-income countries (LMICs), where rigorous social science evidence is no longer scarce but is scarcely utilized, using AI to present data could be a boon for the policymaking community. At CEGA’s recent Africa Evidence Summit conference in Addis Ababa, Ethiopia, we learned that many policy partners globally are regularly turning to LLM chatbots when making decisions; this is particularly worrisome in LMICs where evidence could have the greatest benefit. While LLMs have the potential to help support more evidence-informed policymaking, we are acutely aware of the risks inherent in basing decisions on chatbot text that is unverified, with the potential for misrepresented, low-quality, or even hallucinated evidence.
CEGA is working with our partners at the Global Poverty Research Lab (GPRL) at Northwestern University and Innovations for Poverty Action (IPA) to develop verified, accessible research outputs that use LLMs to assemble large datasets of evidence aggregated from published academic research under the DEVIDENCE project. By aggregating findings and data from academic studies, we are building a public repository of knowledge based on studies of what works, and doesn’t work, in the social sciences. We are currently only working with randomized controlled trials (RCTs) but plan to expand to include other research methods. Ultimately, we hope that the DEVIDENCE dataset will be a free, easy-to-use resource that lets decision-makers quickly compare findings across studies and contexts to identify the most effective interventions.
Aggregating Data through DEVIDENCE
Our joint evidence aggregation project is named DEVIDENCE, for development evidence. DEVIDENCE is a novel effort to systematically link three layers of social science research at scale: study metadata, the results reported in published papers, and the underlying microdata. Metadata maps out where studies took place, the tested interventions, and what outcomes are being measured. The published results give us a measure of how effective these interventions are, and the impact on the outcomes of study. We are working with our partners to link microdata to allow for additional research and to verify the published results. Together, these layers let decisionmakers see what has been studied and at what scale.
Such a platform is critically needed and long overdue. In addition to making study data open and comparable for researchers, it promises to level the playing field for LMIC governments and non-profits who don’t have access to paywalled evidence. We also want to make all of the outputs of the project public so that others can verify our work and add to the DEVIDENCE data.
To conduct this aggregation, the project team built an AI-assisted data extracted pipeline and interface for human data validation. We intend to publish all of the DEVIDENCE data, including our validation approach, in the hope that the data will be a public good for researchers interested in development and metascience. There are many projects creating aggregation platforms. However, users often can’t see how the underlying data were generated – what sets DEVIDENCE apart is transparency about how we generate and verify our data.
Creating a Reproducible LLM Extraction Pipeline
It takes hours to extract all of the treatment effects from a single published economics paper (especially in development economics, where there could be hundreds of treatment effects from multiple experiments). At that pace, creating a repository with granular data from thousands of RCTs would take years of research assistant time. Assuming a modest wage of $25/hour and 10 hours to encode each paper, it would cost $5 million dollars to assemble the data from all ~20,000 impact evaluations published since the 1990s with human coders. LLMs make it possible to build this kind of repository in a matter of weeks or months at a fraction of the cost.
One of the main challenges that we ran into in developing our extraction platform is that findings from social science research are not historically presented in a standardized way. Published papers vary in outcome type, specification methods, heterogeneous treatment effects, and the number of causal claims that they make. Even for the same variable, papers often use different units to measure impacts. Each paper has its own data collection methods and methodology, in addition to individually formatted tables. Before we could begin extracting data, we first took part in a large, multi-institutional effort to create a standardized schema to systematize and harmonize treatment effects in social science research. This schema developed the knowledge framework and definitions that an LLM can understand and apply across multiple papers.
After developing the schema, we shifted our focus to make sure that our LLM pipeline was extracting information in a reproducible way. As probabilistic models, LLMs produce different outputs even if you hold all the inputs constant. We wanted to make sure that our pipeline will extract the same treatment effect information — in other words, how big was the effect of a particular program — from papers if prompted hundreds of times (or with a different model, in the event of deprecation). We took the following steps, which we hope are useful for other researchers encountering similar challenges:
- We use deterministic rather than agentic models in parts of our workflow to try to reduce the variation of outputs. We use a reasoning model only for the steps too complex for a deterministic approach. This also has the added benefit of minimizing costs of the whole pipeline.
- We also give the LLM a schema: a very specific structure of how we want the LLM to output data. While researchers can’t change or orchestrate how the LLM processes information, we can exert control over how it outputs data with very specific instructions.
We tested our LLM against an initial set of 70 papers of hand-coded treatment effects before we began to code more papers at scale. In those initial 70 papers, our team chose papers that had purposefully complicated tables, in an effort to try to address potential errors that could come up from complex papers right away.
Arriving at Human in the Loop Validation
LLMs are great at extracting certain pieces of information, like outcomes, but struggle with more complex pieces of information, like treatment effects. Our team is focusing our validation efforts on improving the extraction of treatment effects. It’s tempting to use another LLM to check variation in LLM data (and our team discussed it). But using another LLM invites the previously discussed challenges about the variation in LLM outputs and opacity of LLM processing.
We’ve concluded that thus far we do need humans in the loop to verify LLM extractions of treatment effect data. For a resource that policymakers will rely on, a single misreported treatment effect could steer resources toward programs that don’t work. Expert verification can make the evidence trustworthy. There is also the added benefit of using the corrections from experts to improve the instructions and schema that we give the model.
The DEVIDENCE Fellows Program
In September, we launched the DEVIDENCE Fellows Program to assist us with data validation. We recruited with a team of scholars, a majority of whom participated in the CEGA Fellowship program, from developing countries to help us verify that the extracted treatment effect information is correct. The initial cohort of DEVIDENCE Fellows are early-career researchers with a background in economics and experimental methods. They check the LLM-extracted data against the original papers, bringing the field expertise needed to judge whether a treatment effect was extracted and interpreted correctly.
DEVIDENCE as a Public Resource
Our hope is that anyone can access the DEVIDENCE portal, quickly learn what is known (and not known) about the cost-effectiveness of development interventions studied in a country and neighboring region in the last 20 years, and confidently make a decision based on this information. We are relying on AI to aggregate data at a scale that was previously impossible, but we are not willing to compromise on data quality. Researchers do still bear the responsibility for providing complete information to other researchers about how we extract and verify data.
We are making exciting progress: by the end of the first Fellows cohort, we aim to have validated data from 1,000 papers, and in the coming months we look forward to releasing our first public dataset. We will continue to share updates on the DEVIDENCE project, as well as the DEVIDENCE fellows program, in the near future.