Machine learning and AI libraries for Python are big. Really big. Nobody wants to download and install multiple copies of PyTorch or CUDA if they can

Search code, repositories, users, issues, pull requests...

submited by
Style Pass
2024-10-31 17:30:07

Machine learning and AI libraries for Python are big. Really big. Nobody wants to download and install multiple copies of PyTorch or CUDA if they can reasonably avoid it.

venvstacks allows you to package Python applications and all their dependencies into a portable, deterministic format, without needing to include copies of these large Python frameworks in every application archive.

It achieves this by using Python's sitecustomize.py environment setup feature to chain together three layers of Python virtual environments:

Application layer environments may include additional unpackaged Python launch modules or packages for invocation with python's -m switch.

While the layers are archived and published separately, their dependency locking is integrated, allowing the application layers to share dependencies installed in the framework layers, and the framework layers to share dependencies installed in the runtime layers.

Alternatively, it can be installed as a user level package (although this may make future Python version upgrades more irritating):

Leave a Comment