Streaming Aggregation and Recording Rules are two ways to tame High Cardinality. What are they? Why do we need them? How are they different?   In this

Streaming Aggregation vs Recording Rules

submited by
Style Pass
2023-05-25 04:00:04

Streaming Aggregation and Recording Rules are two ways to tame High Cardinality. What are they? Why do we need them? How are they different?

In this post, we will cover ways to address high cardinality metrics in time series databases such as Prometheus — one of the most popular open-source time series databases — via Recording Rules and Streaming Aggregations. We will understand the pros and cons of both approaches and their performance impacts. We will cover the following topics.

High cardinality refers to a situation where many unique values exist in a metric data set. In the context of monitoring and observability tools, high cardinality metrics can cause performance and scalability issues.

This means there are 36,000 (10 4 15 * 60) unique combinations, also known as 36,000 TimeSeries, for the http_request_duration_seconds_count metric.

If I add the hostname to the labels and have 100 hosts, the number becomes 3,600,000, or 3 million. This illustrates how quickly cardinality can grow — it multiplies rapidly!

Leave a Comment