Connecting Millions of End-User Devices to your Kafka Pipelines over WebSockets

submited by

Style Pass

2021-05-24 18:00:03

In this article we provide a step-by-step guide to getting started with MigratoryData KE (Kafka Edition), an ultra-scalable Websocket publish/subscribe broker which integrates natively with Apache Kafka. Thanks to its huge vertical scalability (1000x more scalable than the C10K problem) and linear horizontal scalability both for publishers and subscribers, MigratoryData KE can enable bidirectional messaging communication between Apache Kafka and millions of Web and Mobile users or IoT devices - and do so cost-effectively.

We’ve already explained why MigratoryData is a perfect fit for extending Apache Kafka to the Internet, reliably and at scale, as well as the compatibility and complementarity between the two messaging systems.

It is sufficient to say here that both systems MigratoryData and Kafka are publish/subscribe messaging systems, both using a similar notion - named topic in Kafka and subject in MigratoryData - to connect subscribers and publishers. Both Kafka topics and MigratoryData subjects are strings of characters with the following restrictions:

We use a simple convention to dynamically map between Kafka topics and MigratoryData subjects. The first segment of the MigratoryData subject corresponds to a Kafka topic, and the remaining segments of the subject correspond to a Kafka key. Therefore, in MigratoryData KE, the first segment of a subject, corresponding to a Kafka topic, must use at most 255 of the following ASCII characters permitted by the Kafka topics, i.e. [a-zA-Z0-9_-]. The subsequent segments of the subject, corresponding to a Kafka key, can use any UTF-8 characters as there are no restrictions for the keys in Kafka.