Intro to Kafka - Part 1

submited by
Style Pass
2021-06-21 02:30:06

We can write to specific partition using message key and partitioner will map to particular partition using the hash of the key.

Bootstrap.servers , is a list of brokers for producer to connect to with host:port. Recommended setting of 2 brokers for fail-overs.

If we want to have a certain set of messages written to same partition, we can use key and Kafka uses Hash and makes it possible write to same partition.

When multiple consumers subscribed to same topic and belong in same consumer group, each consumer will receive message from diff subset of partition.

If Topic has 4 partitions and consumer group 1 has 1 consumer, then all the messages from all the partitions will be read the consumer 1 from CG1

kafka scales horizontally by adding more consumers to the consumer group. So if we have more consumers in CG ( consumer group ) than partition, some consumers will be idle without receiving any messages.

Adding new consumer to the group or consumer crashing and leaving the group , reading the messages from partition ownership will be changed.

Leave a Comment