Why is observability so expensive?
It’s no secret that observability costs are top of mind for many organizations in the post-zero interest rate phenomenon (ZIRP) era (see here, here, and here for example discussions, though similar sentiments can be found far and wide). Organizations are frustrated with the percentage of infrastructure spend (sometimes > 25%!) allocated towards logging, metrics, and traces, and are struggling to understand how much of this data is actually utilized (i.e., yields real business value) once stored.
Meanwhile vendors are tripping over each other trying to come up with the next great incremental innovation in pricing models and overall cost reduction techniques for storing logs, metrics, and traces. To be clear, there is some innovative and exciting work happening in areas like:
These techniques will possibly lead to better customer experience and lower bills over time. Yet, they are still a small iteration on the basic formula that we have been using for distributed system monitoring and observability for 30 years:
