The hidden cross AZ cost: how we reduced AWS Data Transfer cost by 80%
As expected, AWS services are not for free. And we, as BTB SaaS, are not free either. We definitely try to minimize the infrastructure cost as much as possible to increase the gross profit margin.
When I started my journey as a Cloud Software Architect a few years ago, I heard many stories about huge unnecessary instances that were kept up and running for days or how to leverage spot instances optimization (which I’m not sure are still so economic those days, but this is subject to another article). Everything sounded pretty straightforward: we mainly need to track resource consumption and automate the shutdown of unused ones.
But nothing prepared me for the supersize in our AWS production billing report: the most expensive service, the one that takes 25% of the cost, is cross-AZ data transfer! Not the compute of running a 100 nodes EKS cluster, nor the various types of Databases, while some of them are pretty “managed and expensive” like AppStream, Dynamo, and ElasticSearch, not our MSK Kafka cluster, but the amount of data transferred between AZs within the region…
After talking to our AWS account manager, the assigned solution architect, and the AWS billing expert I realized we are not alone. Many companies suffer from this hidden cost, both in the aspect of not estimating it correctly or struggling with investigation and optimizations. I can think of several reasons why this is so common:
