Having worked on Debezium—an open-source platform for Change Data Capture (CDC)—for several years, one concern I’ve heard repeatedly is this: aren’t you breaking the encapsulation of your application when you expose change event feeds directly from your database? After all, CDC exposes your internal persistent data model to the outside world, which may have unintended consequences, e.g. in terms of data exposure but also when it comes to changes to the schema of your data, which may break downstream consumers.
In this blog post I am going to dive into this problem space, discuss when—and when not—CDC can break encapsulation, whether it matters, and explore strategies for avoiding these problems when it does.
With log-based change data capture—for instance, using Debezium—you can expose realtime change event streams for the tables of your database, sourced from the database’s transaction log. For each executed <span class="inline-code">INSERT</span>, <span class="inline-code">UPDATE</span>, and <span class="inline-code">DELETE</span>, an event is appended to the log, from where it is captured by the CDC tool and propagated to consumers, usually through data streaming platforms such as Apache Kafka or Amazon Kinesis.
These event streams enable a large variety of use cases, such as low-latency data feeds for analytical data stores, cache updates, full-text search indexes, and many more. While there are different alternatives for implementing CDC systems (for instance based on polling for changed rows, or using database triggers), log-based CDC is generally the most powerful and efficient approach and should be preferred whenever possible.