At Mixpanel, we’ve had our ups and downs with the cloud. We moved off the cloud back in 2011 to dedicated hardware deployed on Softlayer. Although w

Evolution of code deployment tools at Mixpanel

submited by

Style Pass

2021-06-11 21:30:07

At Mixpanel, we’ve had our ups and downs with the cloud. We moved off the cloud back in 2011 to dedicated hardware deployed on Softlayer. Although we got rid of some of the noisy neighbor problems we saw on Rackspace, it came at the cost of over-provisioning our processing capacity because it was tough to get machines on-demand. After a lot of soul-searching, we finally decided to move back to the cloud in 2016. At the time, Kubernetes (k8s) was emerging as the industry leader in container orchestration. We really liked Google’s managed k8s offering (GKE), and by late 2017, a large portion of Mixpanel’s infrastructure was running on Google Cloud Platform (GCP).

When we first moved to GKE, we set up two identical clusters. Both were in the same region but in different zones for redundancy and high availability. Each team would deploy services under its own namespace. This gave us a way to scale out our k8s deployments as more and more workloads moved to the cloud. We started with handcrafted YAML files for each of these workloads. Engineers would edit and apply these files ad-hoc. To keep both clusters in sync, they would have to deploy code twice. Occasionally, someone would forget to deploy to the second zone, which led to subtle differences that were hard to debug. Over time, we created a wrapper named mix that would diff the new files against what was running in production to give you an idea of what changes would be applied and then alert you if one zone was out of sync with the other. Soon, mix became the de-facto tool to build and deploy code manually. Say you wanted to deploy a new version of a service foo, you'd run the following commands.

Manual deploys worked surprisingly well while we were getting our services up and running. More and more features were added to mix to interact not just with k8s but also other GCP services. To avoid dealing with raw YAML files directly, we moved our k8s configuration management to Jsonnet. Jsonnet allowed us to add templates for commonly used paradigms and reuse them in different deployments.