By Katie Hoeberling, Senior Program Manager

The Social Science Prediction Platform (SSPP) allows researchers to systematically collect predictions about the results of research, as discussed in this post from last July. When faced with important questions and scarce resources, policymakers and practitioners rely on research and expert perspectives to make decisions (often with implications for many people). However, making good policy decisions often comes down to being able to reasonably predict the effects of policy alternatives, and choosing those that maximize societal benefits.

Since launching nearly ten months ago, the SSPP has welcomed over 1,700 users and 19 projects that have collected over 1,600 predictions. We’re excited to share what we’ve learned about who is using the platform, how researchers want to contribute, collect, or share predictions, and which features are more (or less) helpful. In this post, we’ll get into the nitty gritty of what we’ve learned so far and highlight some particularly useful features for current users and those who are interested but not quite sure where to start.

Who uses the SSPP and how?

Of the SSPP’s 1,743 users, about 48% are graduate students and postdocs, 26% are faculty, and 26% are researchers working for a non-academic institution. Those who have made predictions overwhelmingly represent economics (64%), but psychologists (16%), political scientists (13%), and sociologists (6%) have also taken surveys. This breakdown follows that of the projects, 13 of which focus on economics research, while six focus on psychology or political science topics. Within economics, the development and behavioral fields dominate. Clearly we can diversify our projects and user base much more—we’re excited to work with more psychologists, political scientists, and sociologists over the coming year.

For the most part, researchers have collected predictions of summary statistics and treatment effects. While the SSPP is clearly useful for experimental work, it seems to work just as well for observational research. Arun Advani, Elliott Ash, David Cai, and Imran Rasul, for example, collected economists’ predictions on the prevalence of race-related research in economics journals, showing later that most overestimated these rates. The platform might also be used to collect predictions on how key statistics might change over time.

Emerging benefits of using the SSPP to collect predictions

The list of the SSPP’s possible applications continues to grow—it can be used to gather expert predictions about how an experiment will turn out, to understand common conceptions about a given research question, or simply to be transparent about what our priors are and where they came from, to name just a few. In addition, four major features and benefits have emerged as particularly handy. These have to do with distributing surveys, guidance for creating new surveys, timestamping a project, and sharing results.

1. The SSPP facilitates survey distribution: it makes it both easier for researchers to distribute their surveys and spreads survey requests more equitably across interested forecasters. SSPP projects can select from the following distribution methods to target specific groups or gather predictions from as many people as possible:

• The Prediction Dashboard displays all open, public surveys to anyone with an account. Users can see which surveys they’ve yet to take, are in progress, or have already completed.
• Users can distribute surveys via a custom email sent directly from the platform. This can be especially helpful if you’re trying to restrict your forecaster pool to a specific group.
• Anonymous links can be generated and shared publicly. We’ve used these on Twitter, and seen that they can help alleviate worries that response rates may suffer from people not wanting to sign up for an account.
• Finally, the platform sends out two email digests: one regular newsletter listing all open projects (along with platform updates) and another customized to include surveys in progress or not yet taken. While we’re wary of sending forecasters too many emails, we’ve been pleasantly surprised to see that most users (75%) want to be contacted at least once a month and nearly half want to be contacted more frequently.

2. We’ve created several guides and tools to help develop surveys that are clear to users and get researchers the information they need. In addition to step-by-step instructions on how to submit and distribute surveys, the Survey Guide contains helpful tips for usings Qualtrics, as well as template surveys and annotated surveys you can use as examples.

You can always email support@socialscienceprediction.org for help with issues you can’t find information about.

3. The SSPP provides a timestamped record of your project, with several key dates.
   First, the SSPP provides a record of when forecasts were collected, which is important for those researchers that want to demonstrate that they collected forecasts before the results were known.
   
   Second, the SSPP provides a record of when forecasts were released to the researchers. While researchers will frequently want to view forecasts immediately, there are some situations in which they may wish to wait. For example, if the study is one in which research design choices may affect results, researchers may want to be able to prove they did not see the forecasts before implementing the study and obtaining results (so that the results are unbiased). Similarly, if researchers intend to pre-specify how they will analyze the forecasts but have yet to finish a pre-analysis plan, setting a date in the future may give them some time to post the pre-analysis plan.
   
   Finally, for those researchers concerned about being scooped, the timestamped dates also support the project’s timeline, and the platform offers a suggested citation for each project to further support the provenance of ideas and study designs. If still worried about going public too soon, researchers can always choose to distribute their survey using only custom emails.

4. The SSPP makes it easy to share results from completed projects with those who made predictions. In addition to showing forecasters how their predictions line up with others’, projects can also share results from completed projects to show forecasters how accurate they were.

This is an added incentive for people to make predictions, and can help foster a culture of reciprocity on the platform.

To read more about why we developed the SSPP in the first place or why researchers might want to collect predictions, check out this blog post or visit the Purpose section on the SSPP.

What’s next

The platform will continue to evolve as we learn more about how the social science community wants to use forecasting in research. We invite you to send us feedback and to take the SSPP for a test-drive—sign up or see which surveys are currently collecting predictions here!