DeclareDesign is a set of software packages for planning for and assessing research designs, written by Graeme Blair, Jasper Cooper, Alexander Coppock, and Macartan Humphreys. The software is based on the Model-Inquiry-Data Strategy-Answer Strategy (MIDA) framework for declaring and diagnosing research designs introduced in their paper forthcoming at the American Political Science Review. In this post, the authors of DeclareDesign talk about the tool can be helpful in preparing better pre-analysis plans.

Pre-analysis plans (PAPs) are used by researchers to register a set of analyses before they learn about outcomes. PAPs clarify which choices were made before observing outcomes and which were made afterwards. Ironically, the set of decisions that should be specified in a PAP is itself remarkably unclear. PAP templates and checklists are proliferating, and the number of items they suggest range from nine to sixty. PAPs are becoming longer and more detailed, with some in the American Economic Association (AEA) and Evidence in Governance and Politics (EGAP) study registries reaching hundreds of pages, as researchers seek to be ever more comprehensive. For all that, it is hard to assess whether these detailed plans actually contain key analytically-relevant details.

We propose an approach that clarifies what might belong in a PAP by placing design declaration at the core of a pre-analysis plan. In particular, we propose that PAPs include, in addition to other details:

1. A declaration (in code) of the features of a design.
2. A design diagnosis that analyzes the properties of the design such as power, bias, or coverage via Monte Carlo simulations.

We see these two as related in the sense that the ability to diagnose a design is informative about whether a design declaration is sufficiently complete. This can help researchers assess particular properties of the design.

With some luck, this kind of declaration will make researcher workflows easier. That said, we have just started incorporating design declaration and diagnosis into our own pre-analysis plans, so we are still very much in the process of learning about its costs and benefits.

What is a design declaration?

By “declaration” we mean a formal statement of the questions a design seeks to answer and the strategies for answering them. There should be enough information to simulate the implementation of the design, i.e., it should be possible to specify, for a given possible realization of the world, what the target of inference is and what answer the research design would come up with. In many cases, this means giving information for each of the elements in the “MIDA” (Model-Inquiry-Data Strategy-Answer Strategy) framework:

• A model, M, of how the world works. We are used to specifying expected effect sizes and sample sizes for power calculations, but this leaves out many aspects of the design – the pre-treatment variables and their distributions, expected heterogeneity in effect sizes, the structure of clustering in the data, and other details consequential for assessments of power and other properties.
• An inquiry, I, about the distribution of variables, perhaps given interventions on some variables. The average treatment effect or the average level of support for a political candidate are common inquiries.
• A data strategy, D, is how you as the researcher intervene in the world to change outcomes and generate data. This includes case selection (or sampling decisions) and measurement, as well as interventions such as assignment of treatments.
• An answer strategy, A, that uses data to generate an answer. The answer strategy includes not only regression models and other numerical summaries of the data, but also the tables and figures based on those models.

If all four of these elements are declared, then it can become possible to “run” the design and to see how well it answers questions. If you can do that, we think, then you have provided enough information to remove major analytic ambiguities from a PAP. The fact that you can run the design also demonstrates that you have provided sufficient information about your planned analysis.

We are often especially bad at being precise about what the inquiry is. Informal language can make a question seem clear enough even when it’s not; and the ability to get a precise answer seems to suggest that there must be a well-defined question. But maybe not. As an illustration, we describe a design that seems well-defined, but for which diagnosis reveals that the question itself is poorly formed. Fixing the design means making choices about what the central question really is, which has knock-on implications for other parts of the design.

How might design declaration and diagnosis improve PAPs?

We are obviously enamored of this approach. We see three ways in which declaration can improve the quality of research registration.

1. Declaration can reduce ambiguities in the design. For example, a PAP that says the researchers will “use OLS with clustered standard errors” leaves out which covariates will be included and how they will be coded, whether any interactions will be estimated, and which of several common formulae will be used to calculate standard errors. By contrast, an answer strategy declared in code makes each of these choices explicit.
2. Design declaration can minimize the risk of fishing critiques, since the number of small decisions about modeling and constructing tables and figures that take place after data collection is substantially reduced given the ability to simulate data beforehand.
3. Design declaration may minimize the need for deviations. Simulating the design may reveal aspects of the design researchers were unaware of — variables they did not (yet) plan to collect, assumptions they did not know they needed to make, or planned design strategies were infeasible. The improvements come before implementation not afterwards.

We also see at least three down-stream benefits that can increase the value of registration:

1. Fairer replication, fairer reanalysis. Replicators lose a degree of freedom in determining what an author might have intended by a given analysis. Moreover, ex ante design diagnosis communicates the assumptions under which they thought the design was a good one before they ran the study. This makes it clearer whether a re-analysis involves an improvement in estimation strategies given a researcher’s assumptions or instead requires deviations from a researcher’s assumptions in order to to be justified.
2. Easier design comparison. If a design is declared at the preanalysis plan phase, then it enables direct comparison with the design as implemented in the final write-up. Side-by-side comparison of the code neatly clarifies which design choices were made ex-ante and ex-post. Side-by-side comparison of the performance of a planned and implemented designs clarify under what conditions deviations from plans are defensible improvements.
3. Designs become public goods. With designs formally declared it becomes easier for researchers to take the design code as a starting point to learn from the original investigator’s design choices – or easily modify aspects of them. Declared designs become public goods. We have started a small library of research designs, and we hope you will help us build it up. Our package DesignLibrary allows you to get started with these designs in one line of code.

It’s not (necessarily) all that hard! Design declaration in four languages

Designs can be declared in many coding languages. The DeclareDesign software packages streamlines this process for R users, as illustrated below.

But you don’t have to use our package to do this. To illustrate, we declare the design in three additional code languages: Stata, Python, and via formulae in Excel. We show the process in Excel to demonstrate that design declaration can (in principle) be done in a spreadsheet program – we nevertheless strongly recommend that researchers use a scripting language that transparently reveals design choices. In conducting this exercise, it became clear to us that though declaration and diagnosis are indeed possible in each of these languages, it is not always straightforward. Our software helps in R, but we hope that others will fill in the gaps in Stata, Python, and other languages.

We declare a two-arm randomized-control trial where the goal is to estimate the average treatment effect of an intervention. The declaration clarifies expectations about the effect size and its distribution, the random assignment strategy, and the standard errors used to estimate uncertainty. In the gallery below, click on the declarations to download the code files; the figures to download the code that generated them; and the diagnosis to download a reproducible document that includes the diagnosis. You can also download the code for the reproducible documents for R, Stata, and Python.

Design declaration in four languages

Language	Declaration in code	Figure based on mock data	Diagnosis
R
Stata
Python
Excel

Caveat declarator

We can anticipate some likely objections.

• Writing an analysis plan is time consuming. On the other hand, declaring designs in code could possibly save time since the same code can be written once for analysis planning, execution, and reporting.
• Declaration could make PAPs even more impenetrable if scholars write code in different languages with idiosyncrasies in syntax. Our hope though is that a growth in design declaration will go hand in hand with a convergence around a common “grammar of research designs.” More concretely, with designs formally declared, third parties can interrogate aspects of designs without having to look at the code at all – for example through web interfaces.
• There are real challenges in ensuring that archived code will run over time, which could give declarations a short shelf life. Initiatives like Binder, Code Ocean, nextjournal, and Rocker indicate tools are quickly catching up with these needs.

With all that said, there are likely many objections that we have not considered. We’d love to hear them.

---

Graeme Blair is an Assistant Professor of Political Science at UCLA. He studies the causes and consequences of violence in developing countries with a focus on West Africa. Graeme is a 2016 Leamer-Rosenthal Prize recipient.

Jasper Cooper is a Postdoctoral Research Associate at the Kahneman-Treisman Center for Behavioral Science and Public Policy at Princeton University. He will be an Assistant Professor of Political Science at UC San Diego beginning in July 2019. He studies the comparative politics of developing countries.

Alexander Coppock is an Assistant Professor of Political Science at Yale University. He studies political persuasion and the reproducibility of social science research.

Macartan Humphreys is a Professor of Political Science at Columbia University and a Director of the Research group “Institutions and Political Inequality” at the WZB Berlin Social Science Center. He studies post-conflict development, ethnic politics, political authority and leadership, and democratic decision-making in post-conflict and developing areas.