Learn how Redpanda engineers use Buildkite and GitHub to automatically trigger multiple instances of CI steps running in parallel. At Redpanda, we wan

Using Buildkite for scaling out and running parallel CI steps

submited by
Style Pass
2022-06-23 17:30:02

Learn how Redpanda engineers use Buildkite and GitHub to automatically trigger multiple instances of CI steps running in parallel.

At Redpanda, we want to always provide an experience that is fast, simple, and productive for developers. That applies to our own team of engineers, too. When considering how we could achieve a more stable continuous integration (CI) pipeline, we wanted that same experience: fast, simple, productive. By running multiple instances of our pipeline steps in parallel on our CI platform, Buildkite, we can now run multiple repetitions of the same Buildkite step and use only the amount of time needed for a single step.

Today, our devs can kick off any number of builds in parallel simply by attaching a label to their PR like “ci-repeat-X.” In the rest of this post, I’ll discuss how we made this easy dev experience possible. I discuss how we achieve repeatable builds by taking advantage of Buildkite’s parallelism attribute and pre-command hook, in combination with GitHub labels on pull requests for triggering parallel builds.

When Buildkite introduced a new feature to run multiple repetitions of a build step in parallel, we took advantage of this by adding an attribute in our CI pipeline configuration called parallelism. We use this attribute to define the desired level of parallelism. We started off using a constant value of 1.

Leave a Comment