Engineering At Scale
Modern internet solutions offer a multitude of features for end customers. Generally, these features are built using complex microservices architectures and leverage several storage solutions.
There is no one size fits all solution for data storage. End customers data could be stored in a relational database but a text-search feature would need the same data in Elasticsearch. Similarly, an analytics dashboard would expect the same data to reside in a data warehouse.
A few years ago, I faced a challenge where my team needed data attributes from a system owned by another team. Direct API integration added complexity due to multiple consumers, and a pub-sub setup would have led to redundant effort, as many teams used similar datastores and would have had to duplicate work. This is when I came across Change Data Capture ( CDC) to solve this.
Companies find it challenging to keep the data in one system and store it in different forms in other storage systems. Traditionally, companies used batch processing to Extract, Transform & Load ( ETL) the data in different systems.
