With the recent launch of our Layerfile analytics project, we encountered a cool engineering problem - how to store billions of virtual machine (VM) m

Storing Billions of VM Metrics with TimescaleDB

submited by
Style Pass
2022-09-23 18:00:31

With the recent launch of our Layerfile analytics project, we encountered a cool engineering problem - how to store billions of virtual machine (VM) metrics in PostgreSQL.

With the recent launch of our Layerfile analytics project, we encountered a cool engineering problem - how to store billions of virtual machine (VM) metrics in PostgreSQL. The metrics we wanted to track were the VM's CPU, disk and memory usage at a one second interval throughout the duration of its life. Our data can be considered time-series data as it is timestamped and append only.

While PostgreSQL will be sufficient for our first few million rows, we will start to see its performance decline as the number of writes increases and our table grows. The main pain points we faced included:

TimescaleDB is a time-series database built on top of PostgreSQL. It is an extension to PostgreSQL that implements tools to aide in the handling and processing of time-series data. The Timescale extension runs within a PostgreSQL server processing queries first to determine the planning and execution of queries involving time-series data. This enables much higher data ingestion rates and greatly improves query performance.

Leave a Comment