Clojure: managing throughput with virtual threads

submited by

Style Pass

2024-05-06 16:00:05

Before the introduction of Java 21 virtual threads we were heavy users of claypoole a fantastic library for simple but effective parallelism using thread pools. However, things get complicated with virtual threads, they shouldn't be pooled, as they aren't a scarce resource. So how do we limit throughput when we have "unlimited" threads? In this post we will look at using java Semaphore class to implement basic token bucket rate limiting to control throughput.

One of the other key insights I've drawn from claypoole is that unordered parallelism helps minimising latency, as it allows you to process results as soon as they became available. So we'll also explore unordered parallelism with virtual threads.

We can combine this with an ExecutorCompletionService to get tasks as they complete rather than the order they are submitted in:

We are using Clojure 1.12.0-alpha10 methods as values syntax e.g: ExecutorCompletionService/.submit. This means we don't have to mannualy type hint to avoid reflection.