Databases on Object Storage - the New Normal
When you think about object storage workloads and storage types - databases are not the first thing that comes to mind. That is changing rapidly, however, driven by just two forces: the availability of true, high performance object storage and explosive growth of data and, perhaps more impactfully, its associated metadata.
Because of these two forces, almost every major database vendor now includes S3 compatible endpoints. Further, for many organizations and most workloads, this becomes the default architecture whether in the cloud or on-prem.
The storage performance requirements associated with databases have inverted in the past few years. Databases previously demanded high IOPS. This was a function of the need to make lots of small changes across the network. This was well suited to the SAN and NAS architectures and thus databases became their bread and butter. The problem is that IOPS is not particularly scalable - at least economically.
Databases no longer mutate the data across the network in 4KB chunks. Instead they stream the object (specifically table segments) in MB sized extents to the client-side memory and mutate them locally. The local memory IOPS in combination with a 100GBe makes this a throughput problem - not an IOPS problem.
