Garret Christensen–BITSS Project Scientist

I’m in Barcelona, where I delivered a reproducibility workshop and am attending the International Meeting on Experimental and Behavioral Social Sciences (IMEBESS).  The materials from the workshop are available on Github, as always.

I got several good questions from the workshop participants, some methodological, some software, and I learned a few things, so I thought I’d try and answer them all here.

1. Q: Is reviewing a paper with a pre-analysis plan more work than reviewing one without, since reviewers would now have to compare the results to the design described in the PAP?
   A: Honestly, I’ve never done it, so someone who has should please chime in, but my guess is not really. True, the reviewer should check whether the reported results are in line with the PAP, but to me that doesn’t seem all that time-consuming, and the fact that there’s a PAP might alleviate some of the usual pressure to come up with a novel robustness check that the author hasn’t already tried.
2. The Italian Political Science Review and others organized a pre-registered research competition based around last December’s referendum. This seems very similar to the US Election Research Preacceptance Competition based on the US presidential election. We’re glad the idea is catching on.
3. GitQ: I forked and cloned your repo a couple days ago. How do I make it up to date?
   A: It’s not super straightforward with the Github Desktop app (which is what I use for intro workshops, since I’m trying to keep things simple) but you can follow the instructions here. If you’re command line-savvy, do this. Deleting, re-forking, and re-cloning also works.
4. GitQ: Can I put my Git repo inside my synced Dropbox folder? Using Github Desktop on Mac OS, I get a warning when you try and create a repo inside my Dropbox folder.
   A:  I looked into it, and it seems that if you’re doing simple stuff, and you’re the only one using the repo, you’ll probably be fine, but I make no promises, and you may run into file permission issues if other people edit files. Full disclosure–all my BITSS repos are saved in a Box.com synced folder, and I have never run into problems, though I am typically the only one editing the files. You might consider turning on “selective sync” in your Dropbox preferences (on Windows, right click on the desktop icon, settings, preferences, account, selective sync) and tell it to ignore everything inside your .git folders. Then, of course, your repo’s history isn’t backed up anywhere, and if your hard drive crashes, you’ll still have the latest version in the cloud, but no history (unless you pay extra for Dropbox).

   Generally, saving a repo inside a backup system is a bit redundant, but I talk to a lot of graduate students, postdocs, and RAs who work for professors who don’t want to be bothered learning version control. Github gives you unlimited free public repositories, and students and educators can get free private repositories. If you’re not in that boat, want private repos, and don’t want to pay for them, you can consider using Dropbox as a bare repository. I barely (ha!) know what that means, but you can read about this advanced option here, here, and here. A lot of people seem to use it, but it can fail if you’ve got lot’s of people trying to push/pull simultaneously.

Lastly, I just learned about the checkpoint package in R. It’s designed to automatically run your code using the R packages that existed in the CRAN archive at a specific point in time. So when you release your code publicly, you put in the current date, and then future researchers should be able to reproduce your work exactly, even if packages have changed. I’ve yet to use it, and I don’t know how space efficient it is, but it seems like it may be a lot easier to implement than a docker container.