Like most organizations, we are always trying to be as efficient as possible in our usage of our cloud resources. To help accomplish this, we encourage individual engineering teams at Datadog to look for opportunities to optimize. They can share their performance wins, big or small, in an internal Slack channel along with visualizations and, often, calculations of the resulting annual cost savings. There, these efforts can be seen and recognized by others, including our CEO and CTO, who regularly chime in and comment on performance wins.
At one point an engineer noted that, over the preceding two months, the shared wins added up to $17,500,000 saved annually on our cloud spend. In this post, we’ll look at how our engineering teams achieved this. Specifically, we’ll walk through how we foster an internal engineering culture that builds products that are both excellent and efficient, and how we rely on our own products—including Cloud Cost Management, Continuous Profiler, and Network Performance Monitoring—to track both performance improvements and cost savings so that everyone can see the impact their work has.
In total, there were 15 performance optimizations that accounted for the \$ 17.5 million in annual savings (plotted below), with the lowest optimization yielding \$ 80,000 in annual cost savings and the highest optimization yielding \$ 4.3 million.