Where does the kubelet mount volumes?

Every so often I will be pairing with someone or showing off a demo and realize that some common operation I perform is not well-documented. We all have examples of domain knowledge we learned somewhere along the way and stashed in our toolbox. Whenever I encounter these situations I try to remember to post about them so that the next person can find the information at least a little bit faster than I did.

Recently, I was troubleshooting an issue where some files were being mounted as a volume to a container in a Pod on a Kubernetes cluster. The volume appeared to be mounted successfully, but the container was failing to authenticate with credentials that should have been present in the mounted directory. I wanted to validate that the credentials were in fact valid, but this particular container was using a minimal OCI image (shout out distroless 👏), so using kubectl exec to get a shell was not possible.

At this point I fell back to the familiar process of accessing the Node to see what was going on. Kubernetes can sometimes feel like an opaque API, but it’s helpful to remember that behind the scenes everything is still processes and files on physical hardware with varying levels of isolation. If we quickly spin up a kind cluster, we can start to explore a node that itself is a container running on our local machine.

Leave a Comment