Running a Billion Workflows a month with Netflix Conductor

Netflix Conductor is a well known platform for service orchestration allowing developers to build stateful applications in the cloud without having to worry about resiliency, fault tolerance and scale. Conductor allows building stateful applications — automation as well as human actor driven flows by composing services that are mostly stateless.

At Orkes, we regularly scale Conductor deployments to scale to handle millions of workflows per day and we often get questions from the community on how to scale the Conductor and if the system can be scaled to handle billions of workflows

We ran a benchmark to test the limits of Conductor’s scalability using a simple single node setup on a laptop. The results were not so surprising, given our experience with Conductor, — with the right configuration, Conductor can be scaled to handle the demands of really large workloads.

We used wrk2, a fantastic tool to generate stable load on the server. Wrk2 improves on wrk and adds the ability to generate sustained load at a specific rate (-R parameter).

Leave a Comment