The Many Problems with Celery

submited by
Style Pass
2023-05-21 15:30:04

Sometimes called transactionally staged jobs, or the transactional outbox pattern, but the gist is you can’t enqueue a job inside a transaction.

This means you can’t have an HTTP request handler that creates a new User and schedules a job to send them an email, because the database save might succeed but the job might not persist, or vice versa.

Since Celery doesn’t have transactional job enqueuing, canvas, chords and friends are a recipe for losing jobs or having broken workflows.

If you want to access Celery’s builtin retry methods (along with additional context) you need to use the bind kwarg in your task definition, which causes Celery to pass in a first argument automatically. This behavior is tricky to type check and results in an inconsistent API.

The problem is you have existing workers running with the old task version, and existing jobs serialized in the queue, which are incompatible with the new task version, so you end up with a bunch of exceptions.

Leave a Comment