Opsqueue: a lightweight batch processing queue for the heaviest loads — now open-source!

opsqueue is a good choice if you have a use case where you first generate a few million operations (an "operation" is any task that can be executed within a few seconds) and then later execute those operations. In our case, and we expect many other use-cases as well, the systems which generate the operations are often different from the systems that execute them. We usually want to have specialized consumers that maximize throughput (within certain constraints, like e.g. rate-limits) by processing operations in parallel.

There are several reasons why service A often doesn't apply the operations itself, and instead offloads them to another service. Usually service A is either not scalable enough, or it is not specialized to do the necessary task, like e.g. calling an LLM service, editing an image, or talking to an external API with strict rate-limits. By offloading these tasks to more specialized services we can optimize them for each specific task.

We call services that place input into the queue 'Producers', and services that take work from the input queue and write results back to the output queue 'Consumers'. The description of the transformation to do for a particular element we call an 'Operation'. This word is chosen rather than a more general term like 'task' because individual operations ought to take milliseconds to a couple of seconds at most. opsqueue is thus a queue for operations.

Leave a Comment