But instead of having to hard-code node names and adding receive messages sent by spawned processes, I wanted a slightly nicer API. Instead of specify

Implementing distributed pooling in Elixir

submited by
Style Pass
2024-11-06 13:00:07

But instead of having to hard-code node names and adding receive messages sent by spawned processes, I wanted a slightly nicer API.

Instead of specifying node names(like :"another_beam_instance@10.0.1.2" as we did in our example, with FLAME you specify a pool name(MyApp.FFMpegRunner). The function in the second argument then runs on some node that's in that pool. It's very much like a GenServer call– the calling code doesn't have to know that it's running on a different machine.

So, I was trying to put together something that looks like the FLAME API but rather than FLAME spinning up machines, the idea was to let existing nodes in your cluster join pools.

For a given pool(call it the FFMpegPool ), the idea is that nodes that are part of the pool start a DynamicSupervisor This DynamicSupervisor would then join a process group(using pg) with its fellow DynamicSupervisors in the pool.

We use pg because this process group spans our cluster– so we can create a list of each DynamicSupervisor that we started for the FFMpegPool.

Leave a Comment