Python has too many package managers

Python is a wonderful programming language. I’ve used it to build webapps, deep learning models, games, and numerical computation. However there is one aspect of Python that has been an inexcusable pain-in-the ass over many years. That would be the fragmented Python package and environment management ecosystem, succinctly represented by the following XKCD comic:

You see, a lot of other programming languages developed standardized ways to setup versioning, dependency resolution, and dev environment setup. C# has NuGet, Javascript has npm, Dart has pub, and most notably Rust has Cargo – quite possibly the most widely loved package manager tool in existence.

In a sane world, package management would work like it does with Cargo - the rust package manager. You have a single master configuration TOML file where you simply list your dependencies and config settings. The TOML file goes into a folder encapsulating your entire development environment. For extra reproducibility, whenever you build your environment and resolve all your package dependencies, a *.lock file records all the packages you used along with their versions and hashes.

Finally, because dependency resolution is a directed acylic graph (DAG) resolution problem, the dependency retrieval and resolution should both be engineered to be relatively fast. Dependency information should be freely available from a public API metadata server in a way that is simple to parse, and cached locally once downloaded to avoid redundantly hitting this server. Finally, the dependency resolution algorithm itself should be written in a relative fast programming language like C++ or Rust.

Leave a Comment