Producer and consumer reliability
In distributed systems, components fail. Its a common practice to design your code to take care of these failures in a seamless fashion (fault-tolerant).
One of the ways by which Kafka tolerates failure is by maintaining the replication of messages. Messages are replicated in so called partitions and Kafka automatically elects one partition as leader and other follower partitions just replicate the leader. The leader also maintains a list of replicas which are in sync so as to make sure that ideal replication is maintained to handle failures.
The producer sends message to the topic (Kafka broker in Kafka cluster) and durability can be configured using the producer configuration, request.required.acks, which has the following values:
- 0: message written to network/buffer
- 1: message written directly to partition leader
- all: producer gets an acknowledgement when all in-sync replicas (ISR’s) get the message
Consumer reads data from topics and in Kafka the state of...
