Service Monitoring Patterns
Hotstar as a platform deals with more than 1M RPS of pure API calls from various clients (this excludes calls for images/video). At peak, the volumes go up to 5M RPS. Hotstar is a global platform with more than 200+ micro-services working together to give our customers a smooth experience.
In this pattern, each of the systems running a service is emitting metrics from within, the source of the truth is the system itself that is being monitored, this typically means that you are trusting that all will be well with the system even when it is stressed out and would send the distress signal as well.
This approach basically assumes that every component in your platform can fail, and hence you can’t trust the data given by it. That’s why you always ask the layer above it, which is calling it to provide data on how this system is performing, and you do this all up in the chain.
Our tech stack varies by use-cases. We have a variety of data stores used across the platform. Some of these are fully managed services and some are managed by us. Unfortunately, there is no uniform monitoring solution for these, each one of them emits the metrics in their own way.
