All You Need Is S3 |

submited by

Style Pass

2024-02-12 12:30:04

Over the past decade, the tech industry has significantly invested in the development of the cloud-native ecosystem, pouring vast amounts of human capital into its expansion. Despite the apparent diversity, a common underlying challenge unifies most projects: achieving consistent data sharing between processes.

The term “data” encompasses a broad spectrum, from files and messages to database records and event streams. The current dominant paradigm has been to partition the task of sharing data into distinct domains, such as object storage, message queues, and databases. These domain specific systems invariably feature a half baked implementation of Paxos [1][2], a custom persistence layer, and bespoke APIs.

However, at its core, data remains merely a sequence of bytes. Do we really need to slice it, dice it, and treat it so differently?

Admittedly, each system is optimized for specific data types, employing various optimizations and transformations (e.g., indexing, compression, etc) before persistence. But when it comes time to flush the data to disk, the target is invariably a file sitting on a filesystem. So it’s pretty clear that we all implicitly agree that when it comes to durabilitiy, POSIX filesystems are “good” enough [3].