The problems with (Python's) Celery

Disclosure: I'm a co-founder of Hatchet (opens in a new tab) , a multi-language task queue with Python support. We're open-source (opens in a new tab) (with a cloud version) and we aim to be a drop-in replacement for Celery that supports a modern Python stack.

Most people who build web apps in Python have encountered Celery (opens in a new tab) . It's described as a “distributed task queue” — which basically means that it helps users divide background tasks between a set of processes, called workers.

And it's widely deployed for good reason: it integrates easily into nearly every Python framework, supports a wide range of brokers and backends, and has a number of customization options and higher-order concepts like chains and chords which make it a compelling tool.

While Celery has gotten a lot right, there are many potential pitfalls when using Celery. These come from personal experience — both on the infrastructure and application side as heavy Celery users — along with migrating a number of Celery users to Hatchet. We've grouped these issues into 3 categories:

Leave a Comment