Exploring the Kafka architecture
Kafka has a distributed architecture that consists of brokers, producers, consumers, topics, partitions, and replicas. At a high level, producers publish messages to topics, brokers receive those messages and store them in partitions, and consumers subscribe to topics and process the messages that are published to them.
Kafka relies on an external coordination service called Zookeeper, which helps manage the Kafka cluster. Zookeeper helps with controller election – selecting a broker to be the cluster controller. The controller is responsible for administrative operations such as assigning partitions to brokers and monitoring for broker failures. Zookeeper also helps brokers coordinate among themselves for operations such as leader election for partitions.
Kafka brokers are the main components of a Kafka cluster and handle all read/write requests from producers/consumers. Brokers receive messages from producers and expose data to consumers...
