Zero Disk Architecture

submited by
Style Pass
2024-11-24 15:30:06

In my previous post, I explained how a disk attached to a machine makes things difficult. Vertical scaling has its limits, and when you hit that limit, you can’t do horizontal scaling right away because of the attached disk. Mainstream databases like Postgres or MySQL don’t scale horizontally. I recently learned that BlueSky team switched from Postgres to a combination of Scylla and SQLite. One of the reasons was because (vanilla) Postgres is not horizontally scalable, but Scylla is.

State is pain. Since the machine is stateful, you lose elasticity and scalability. So, the solution was to separate state from compute, so that they become independently scalable.

But there was a big cliffhanger at the end of the post. The storage server. If I am writing a storage server, then won’t I need to manage the state? It looks like we are back where we started. We need a storage server which is strongly consistent, elastic, horizontally scalable, and preferably has auto sharding.

In large companies, you can offload the storage server problem to another team and live peacefully. For example, Amazon has a transaction log service (the details aren’t public) which is used by Aurora and MemoryKV.

Leave a Comment