Starting with Kafka · Tinybird

I just want to share my thoughts on Kafka after using it for a few months, always from a practical point of view. I don’t know anything more than the basics about Kafka internals. I knew the principles of log processing and so on but had never got my hands dirty. So this is just the opinion of a newcomer, not an expert.

Everything began because we started seeing event-gathering use cases: a website, with a reasonably high amount of traffic that wants to gather events, to measure in real-time what’s going on. Like a customized version of Google analytics but without sampling. We’ve spent the last few months working on an integration with Kafka to cover these use cases.

I have to admit that I was biased against Kafka, I thought it was a large, complex piece of technology that added nothing but an extra layer of complexity.

I still think the same, it is complex, adding Kafka to your stack will add a bunch of services and some extra decisions to make about configuration and so on. In most cases, a single process app sending data to a file and a small HTTP server serving it will perform much better with less downtime. If you compare the burden of understanding, setting-up, configuring and starting to use compared to, for example, Redis, it’s a massive amount of work.

Leave a Comment