How to take down production with a single Helm command

submited by

Style Pass

2024-04-04 05:00:06

No worries. You’re using Helm. You can roll this change back safely. You ask your colleague. “Oh yeah, helm rollback should work.”

After a hectic troubleshooting session with the team, you redeploy the service and start investigating. A colleague uses the staging environment to do a helm rollback and it works as expected, the previous version of the service is successfully deployed.

You investigate logs. The helm rollback call worked as expected, and then it began deleting every entity related to the deployment. Pods, secrets, ingresses, everything related to the service was gone, and your name was present on each deletion.

The troubleshooting was on standby for a few days since you had no further leads and had to get other work done. But you couldn’t really move on from this issue mentally, could you?

One day you continue the investigation by opening the Helm GitHub repository, looking at the open issues and throwing in some keywords that might be relevant, such as “rollback”.