Book-Driven Development from “Boost Your Django DX”
On Monday I released my new book “Boost Your Django DX”. It covers many tools and practices that are useful for developing Django projects.
Whilst writing I often try to describe something and find it hard to explain, or suboptimal in some way. Since I want to make things as easy as possible for readers, this process leads me to make open source improvements, or new packages. I have taken to calling this process Book-Driven Development.
Here’s a list of the more interesting packages and contributions I made whilst writing this book.New package: django-browser-reloadChapter 5 of the book covers ways to enhance Django’s development server, runserver. Whilst writing it, I took some time to research tools for automatically reloading the web browser files change. I knew of some tools out there, I had just never seen any that were easy to set up. Most relied on running an extra Node.js process and proxy server, which felt like way too much for my tastes.After a little bit of digging, I decided to try my hand at writing my own browser reloader. My proof-of-concept worked so well that I spent the next day turning it into a package. And so, I created django-browser-reload.The package only relies on tools built in to Django and vanilla JavaScript. This made it easy for me to recommend as a generally applicable tool in my book. I hope its simplicity makes it accessible.You can read a bit more about django-browser-reload in my introductory post.After releasing the package I found that frequent Django contributor Keryn Knight has a similar package called django-livereloadish. It’s a bit more experimental, but there are some ideas I’d like to merge into django-browser-reload, such as saving and restoring page state.New package: django-richRich is a wildly popular package for pretty terminal output. In the book I describe using it with Django, both for logging and within management commands.The logging use case is easy enough, as Rich seamlessly integrates with the logging module. Django configures logging with its LOGGING setting.Management commands are a little trickier though. In command code, it’s best to use Django’s stdout and stderr wrappers, rather than calling print() directly. These wrappers improve testability: Django’s call_command() lets you use them to to capture output for assertions.I started writing a section on using Rich’s Console class with the output wrappers, but found that covering all the cases is a bit involved. It was too much code to tell readers to copy and paste, so I put the code into a new package, django-rich. The package provides a command class that sets up the Console for you: from django_rich.management import RichCommand class Command ( RichCommand ): def handle ( self , * args , ** options ): self . console . print ( "[bold red]Alert![/bold red]" ) Easy peasy!The package may grow to cover more ways of using Rich with Django. I wrote a couple of ideas in its issues list.New package: django-upgradeBefore I started properly working on the book, I was thinking about tools to improve DX. I really enjoy using pyupgrade, which rewrites code using old Python syntax to new. It’s handy when run as a pre-commit hook, as it can then update new code added that accidentally uses old features.django-codemod is a similar tool for upgrading code using old Django features to new ones. I’d used django-codemod and it works great. But it is unfortunately slow—it could take two minutes to process a medium-sized project, with all CPU cores a-blazing. It is therefore too slow to practically run as a pre-commit hook.I tried experimenting with rebuilding django-codemod with the same underlying approach as pyupgrade. This worked well, so I turned the experiment into django-upgrade. I since added more fixers, and released the tool, and it’s been fairly popular.You can read a bit more about django-upgrade in my introductory post. The book covers both pyupgrade and django-upgrade.New package: pre-commit-oxipngI want to keep my book as small as possible. The largest part of its file size, in any format, is the images. Most of these are PNG screenshots demonstrating.PNG’s can be easily optimized, especially my screenshots (macOS seems to prioritize quick saving over a smaller filesize). Often the savings are more than 50%!I’ve previously used the pngcrush tool to optimize PNG’s, but it’s a little “clunky” (difficult CLI, no parallelization). I came across oxipng as a modern, Rust-based alternative. It’s fantastic!I wanted an easy way to run oxipng under pre-commit, and so I created repository pre-commit-oxipng. This repository contains the appropriate pre-commit configuration, and it automatically syncs oxipng releases thanks to pre-commit-mirror-maker. This way there’s an easy public repository for shared configuration, but the oxipng repository doesn’t need to be concerned with pre-commit. (I still opened an oxipng issue to ask if they’d like to host the config.)Contributions: django-debug-toolbar Improvementsdjango-debug-toolbar is a fantastic tool for development. It’s very popular: according to the 2021 survey, 26% of Django developers put it in their “top five” favourite packages.My book describes setting up the toolbar and using it for several common tasks. Whilst writing this section, I found myself looking at the package’s installation instructions and source code, for the first time in a while. I made several PR’s based on this exploration:PR’s #1533 and #1534 improved the documentation. I found several ways to improve the writing to make it clearer and briefer.PR’s #1571 and #1574 optimize the SQL panel processing. For an N+1 queries page in my test app, these two changes add up to ~30% reduction in total run time. I know some developers can’t keep the toolbar always-on because of its overhead, so hopefully this optimization helps in those situations. From looking at profiles and code, there is definitely more room for optimization.PR #1535 reduced the ways of running Flake8 and Black to only pre-commit. Previously the repository also had configuration to run these tools with make and tox. I think this was a case of developers adding alternatives over the years but never pruning them. These alternatives had the possibility of installing different versions, which could cause code inconsistencies. pre-commit is a great tool for running code quality tools, and it has wide adoption in the Python community. There are three chapters in my book covering setting up pre-commit, using it to run various tools, and even writing custom tools.PR #1559 moved the “settings reset” function to the package’s settings module. Thisi function receives Django’s setting_changed signal and uses it to clear caches based on setting values: from django.dispatch import receiver from django.test.signals import setting_changed ... @receiver ( setting_changed ) def update_toolbar_config ( * , setting , ** kwargs ): """ Refresh configuration when overriding settings. """ if setting == "DEBUG_TOOLBAR_CONFIG" : get_config . cache_clear () elif setting == "DEBUG_TOOLBAR_PANELS" : ... Settings may change during tests when using e.g. the @override_settings decorator. Previously this function lived only in the django-debug-toolbar test suite. That is fine for the toolbar’s own test suite, but it meant it wouldn’t apply in test suites of projects using the toolbar. By placing the reset function within the main package, the settings are reset whenever they’re changed, not just within the package tests. This follows the same pattern that Django uses.PR #1560 avoided injecting the toolbar HTML when the response Content-Encoding header is set. The toolbar tries to inject its HTML into all text/html responses. Previously it avoided doing so if the response had the header Content-Encoding: gzip. gzip isn’t the only possible encoding though, there’s also e.g. br for Brotli. The PR changed the logic to avoid injecting the HTML whenver the response has any encoding, because it cannot decompress the contents. (In most setups, responses will be compressed after the toolbar HTML is injected—the settings even guide you to do so.)There are still more ways to improve django-debug-toolbar, so some more PR’s may be forthcoming soon!Contribution: sphinxcontrib-spelling OptimizationI write my book with Sphinx, the primary documentation tool in the Python ecosystem. It’s used to document Python, Django, and most projects on Read the Docs.To check my spelling, I use the sphinxcontrib-spelling extension. I found out about this package because it’s used for the Django docs.To run the spell check automatically, I run it with pre-commit. This prevents me from even committing misspelled words. Unfortunately I found that as I wrote more, this process took longer and longer.I profiled the spell check process, and discovered that most of the time was spent in a class called ImportableModuleFilter. This class checks if each word names an importable Python module, and if so, prevents the word being counted as a spelling mistake. After reading the source I found some optimizations that I made in PR #129.The optimizations helped, but only reduced the runtime by about 10%. So I decided to disable several of the spell check filters, with these settings: spelling_lang = "en_GB" tokenizer_lang = "en_GB" spelling_ignore_acronyms = False spelling_ignore_contributor_names = False spelling_ignore_importable_modules = False spelling_ignore_wiki_words = False This change took more than 50% off the runtime. The only downside is that when mentioning importable modules for the first time, I have to add them to by spelling “allow list” file. I find that pretty acceptable.FinThanks to all who buy my book—it really helps support this development,—Adam
Chapter 5 of the book covers ways to enhance Django’s development server, runserver. Whilst writing it, I took some time to research tools for automatically reloading the web browser files change. I knew of some tools out there, I had just never seen any that were easy to set up. Most relied on running an extra Node.js process and proxy server, which felt like way too much for my tastes.
