Kube-proxy and mysterious DNS timeout | Charles Xu
This post reviews how iptables-mode kube-proxy works, why some DNS requests to kube-dns were blackholed, and how to mitigate the issue.
The kube-dns Service uses a label selector to select all CoreDNS Pods. The Service has a ClusterIP. Requests to such ClusterIP will be DNAT-ed to one of the CoreDNS Pod IPs. The DNAT is performed by kube-proxy, which runs as a DaemonSet. Kube-proxy is not a real proxy (data plane) but configures the iptables rules and conntrack tables on the Node to implement the DNAT.
DNS is primarily over UDP. Although UDP is a connectionless protocol, kube-proxy still uses conntrack for UDP to remember the NAT translations applied to each pair of the source and destination IP addresses and ports, ensuring that responses can be correctly routed back to the originating Pod.
When the CoreDNS Deployment had a rolling restart, new CoreDNS Pods had new IPs, and old CoreDNS Pods were removed so their IPs became stale. Thus, kube-proxy needs to update the Node’s iptables rules and conntrack tables.
