Replication package for "Improving out-of-population prediction: The complementary effects of model assistance and judgmental bootstrapping"

This repo contains analysis code, experiment code, and data for Improving out-of-population prediction: The complementary effects of model assistance and judgmental bootstrapping by Matt Hardy, Sam Zhang, Jessica Hullman, Jake Hofman, and Dan Goldstein.

This reproducibility package was assembled on September 9th, 2024. For questions, contact Matt Hardy at [email protected].

Contents

Scripts

The data_analysis directory contains the following R Markdown files:

• experiment_results.Rmd: Runs all the statistical tests cited in the paper. Our main analyses and sample sizes were preregistered. The results of each analysis (H1-H4) are printed in the final code block. Runtime: ~30 seconds.
• final_experiment_figures.Rmd: Generates the plots included in the paper. Runtime: ~60 seconds.
• multi-choice-utils.R: Contains helper functions used in the analysis scripts (regular R script).

Data

The experiment_data/ directory contains:

• raw_experiment_data.csv: Raw experiment data
• experiment_data_with_bootstrap.csv: Experiment data with fitted bootstrapped estimates
• worker_data.csv: Participants' responses to the comprehension quiz
• monthly_averages/: True high temperatures for target and model cities

Experiment Code

The experiment/ directory contains the front-end code and data for running the experiment. A demo is available here.

Key files:

• experiment_with_city_selection.js: Main experiment script (jsPsych)
• model_trial_with_selection.js: Custom jsPsych plugin for trials
• city_data.csv: Data for target city selection
• us-states.geojson: Data for USA map state outlines
• custom.css and index.html: Styling and main HTML file
• comprehension_questions.pdf: Comprehension quiz questions and answers

Note: To run the experiment locally, launch a server in the experiment directory.

Setup and Reproduction

To run the analyses:

1. Ensure you have R (version 4.0.0 or later) and RStudio installed.
2. Clone or download this repository.
3. Install required packages:

   install.packages("remotes")
   
   packages <- c("dplyr", "emmeans", "lme4", "lmerTest", "here", "ltm", "glmnet", "ggplot2", "stringi", "lubridate", "ggpattern", "tidyr")
   versions <- c("1.0.8", "1.8.1.1", "1.1.28", "3.1.3", "1.0.1", "1.2.0", "4.1.4", "3.4.4", "1.7.6", "1.9.2", "1.0.1", "1.2.0")
   
   for (i in seq_along(packages)) {
     remotes::install_version(packages[i], version = versions[i])
   }

4. Open the desired .Rmd file in RStudio.
5. Click "Knit" to run the analysis and generate the output document.

Each R Markdown file will:

1. Load required packages
2. Set up the working environment using the here package
3. Source 'multi-choice-utils.R' for additional functions
4. Perform analyses or generate figures