PaperOps: run experiments and add results to a PDF with a single Make command

submited by

Style Pass

2024-05-10 15:30:36

In my work as research faculty and a PhD supervisor, I constantly try to streamline the research process. Much of that work involves automating running (or re-running) of experiments so that we can quickly test new hypotheses, change parameters, or even just change colors in a figure.

My most recent project has been to develop a simple proof-of-concept in which every aspect of running results to including them in a paper is automated. Simply download the repo from GitHub, run make paper, and it outputs a PDF file with the results from some simple experiments.

This simple command runs 25 “experiments”—a simple example task that fits a line to noisy data with and without outlier rejection and compares the two—and generates a plot that contains a scatter plot of those results and computes statistics for the results. This results in the following PDF, manually cropped to highlight the important bits:

Make paper: 25 experiments. The PDF includes the correct number of experiments, statistics, and a scaterplot of those results, all via make paper NUM_EXPERIMENTS=25. Make paper: 25 experiments. The PDF includes the correct number of experiments, statistics, and a scaterplot of those results, all via make paper NUM_EXPERIMENTS=25.