How GoodJob's Cron does distributed locks

submited by

Style Pass

2023-01-23 11:00:12

GoodJob is a multithreaded, Postgres-based, ActiveJob backend for Ruby on Rails. GoodJob has many features that take it beyond ActiveJob. One such feature is cron-like functionality that allows scheduling repeated jobs on a fixed schedule.

This post is a brief technical story of how GoodJob prevents duplicated cron jobs from running in a multi-process, distributed environment.

GoodJob heavily leans on Concurrent::Ruby high-level primitives, and the cron implementation is no different. GoodJob::CronManager accepts a fixed hash of schedule configuration and feeds them into Concurrent::ScheduledTasks, which then trigger perform_later on the job classes at the prescribed times.

A locking strategy is necessary. GoodJob can be running across multiple processes, across numerous isolated servers or containers, in one application. GoodJob should guarantee that at the scheduled time, only a single scheduled job is enqueued.

When GoodJob’s cron feature was first introduced in version 1.12, Cron used an existing feature of GoodJob: Concurrency Control. Concurrency Control places limits on how many jobs can be enqueued or performed at the same time.