Testing web-based notebooks

submited by

Style Pass

2021-07-31 22:30:04

Notebook systems for literate programming are increasingly popular. I’m talking about tools that allow you to mix code, text and rich content like Jupyter, Observable, and Starboard (which I’m building).

For Starboard I wanted to be able to make changes to Starboard itself and plugins while being confident everything still works end to end.

The most serious tool I could find for Jupyter is testbook (part of the nteract ecosystem). This tool seems to follow a pattern that I saw used in other Jupyter testing setups: notebooks are treated a .py files, which allows us to test it as any other Python file.

This feels wrong, why not put this function in a separate file anyway and import it into the notebook? In my opinon, a notebook should present a narrative, serve as your experimentation bed, or allow the user to learn or do something.

What if we want to test the actual notebook itself and not just its individual functions? What does that even mean? In short: I want to ensure I can run all the cells in the notebook start to end without any uncaught errors.