As we have mentioned in our previous posts, Beat’s architecture consists of multiple islands that each one of them contains a separate Kubernetes c

Sometimes It’s Not Always DNS

submited by

Style Pass

2021-07-18 11:30:12

As we have mentioned in our previous posts, Beat’s architecture consists of multiple islands that each one of them contains a separate Kubernetes cluster. Recently we took the decision to allow services from different clusters to be able to discover each other. Since we were already using Istio we decided to leverage Istio’s multi cluster capability.

After preparing our setup and testing it with some of our home grown tools, we tried to switch one of our lower traffic services to use Istio’s multi cluster discovery. Not long after enabling it, we got reports from some of our fellow engineers that there were some issues with it. Occasionally, services from one cluster were failing to resolve services from a different cluster. The DNS resolution errors were not consistent and were only for endpoints that belong to different clusters.

Before jumping into our case, it might be useful to give some background about Istio’s multi cluster architecture. Istio’s multi cluster implementation is heavily based on DNS and a high level idea is given in one of Istio’s own articles.