A CloudWatch Bill Adventure

AWS bills are hard to dig through. My team stumbled upon another weird artefact we can't make sense of. For the first five days of a month we have twice the CloudWatch costs compared to the rest of the month.

Let's use the tools given to us by dear Jeff and drill down on this thing. Grouping by usage type gives us a hint. The cost originates from “EUC1-CW:MetricMonitorUsage”. The name very clearly points us in the right direction: We accumulate costs through custom CloudWatch metrics. These are metrics outside of the AWS namespace.

We get a better view of things if we group by API operation. We see “MetricStorage” doubling and, if you squint enough, “MetricStorage:AWS/EC2” also roughly doubles. This lead me to believe, that we have a custom metric which we write more often at the start of the month. The sample count of the metric should be higher for these first few days.

We want to check the sample count of each CloudWatch metric. A little script helps listing all metrics. It then fetches and accumulates the sample counts. With a CSV in hand, I headed over to ObservableHQ to do some data exploration. A heat map should help us highlight whether we write a given metric more often. Here is a small excerpt:

Leave a Comment