Building a custom Envoy discovery service for Depot
When running inside AWS, we host your BuildKit instance behind a public IPv4 IP address and connect you directly to the instance via an on-demand mTLS certificate we issue per build.
However, when outside of AWS and inside of something like Fly.io, we need to do the same kind of routing but without the ability to have a public IP address.
To do this, we need to route the BuildKit TCP traffic based on the TLS Server Name Indication (SNI) and forward it to the correct host. Each host has its own unique mutual TLS (mTLS) configuration for a given build.
In this post, we will discuss how we built a custom Envoy discovery service for Depot to route traffic based on SNI to the correct BuildKit instances running on Fly.io.
Each Docker build project in Depot is a BuildKit instance for each architecture you're building for. When you run depot build, we create a new BuildKit instance on-demand and connect you to it via a secure mTLS connection. If you run depot build --platform linux/amd64,linux/arm64, we create two BuildKit instances, one for each platform.
As mentioned above, inside AWS, we connect you to the public IPv4 address of the builder instance. We issue a unique mTLS certificate for each build and connect you to the instance via that certificate. It's rather straightforward, as we just route the build to the builder's public IP.
