InfoQ Homepage Articles Tales of Kafka at Cloudflare: Lessons Learnt on the Way to 1 Trill

Tales of Kafka at Cloudflare: Lessons Learnt on the Way to 1 Trillion Messages

submited by

Style Pass

2023-05-30 10:30:04

InfoQ Homepage Articles Tales of Kafka at Cloudflare: Lessons Learnt on the Way to 1 Trillion Messages

Cloudflare has generated over 1 trillion messages to Kafka in less than six years just for inter-service communication. As the company and application services team grew, they had to adapt their tooling to continue delivering fast.

We will discuss the early days of working in distributed domain-based teams and how abstractions were built on top of Kafka to reach the 1 trillion message mark.

ScyllaDB is the database for data-intensive apps requiring high throughput + low latency. Achieve extreme scale with the lowest TCO. Learn More.

We will also cover real incidents faced in recent years due to scalability limitations and the steps and patterns applied to deal with increasing demand.

Cloudflare offers a range of products, including CDN, Zero Trust, and Cloudflare Workers, to achieve these goals, identifying and blocking malicious activity and allowing customers to focus on their work.