Imagine you’re running a large coffee beans retailer, with a warehouse full of various grades of beans. One day, there is a spike for demand, where

Lowering Latency by Moving From Kafka to Redpanda

submited by

Style Pass

2021-07-08 22:00:10

Imagine you’re running a large coffee beans retailer, with a warehouse full of various grades of beans. One day, there is a spike for demand, where everyone buys every single variety of coffee. In a short time, chaos ensues in the warehouse team, not knowing what kind of beans are left. All of a sudden, hundreds of your clients ordered the grade A coffee at the nearly the same time, with your team not knowing who would have the right for the limited amount of those grade A beans. These kinds of problems are even more profound in large-scale, big-data style companies, where billions of data are generated every second from some source and need to be further processed downstream by another service, such as the order and warehouse in our previous example. The question is, how do companies track and deliver these data in time, without any attenuation along the way, and with precision?

Technically speaking, there are some methods to try and send data over, the simplest of which is using manual database, either through pen-and-paper, or some kind of spreadsheet applications then sending it through snail mail or email. This has worked since the beginning of society, however there are some glaring problems besides the obvious speed of the transfer of information; data loss, scalability.