Most of you must have used the LinkedIn portal in your professional career. The Kafka system was first built by the LinkedIn technical team. LinkedIn constructed a software metrics collecting system using custom in-house components with some support from existing open source tools. The system was used to collect user activity data on their portal. They use this activity data to show relevant information to each respective user on their web portal. The system was originally built as a traditional XML-based logging service, which was later processed using different Extract Transform Load (ETL) tools. However, this arrangement did not work well for a long time. They started running into various problems. To solve these problems, they built a system called Kafka.

LinkedIn built Kafka as a distributed, fault-tolerant, publish/subscribe system. It records messages organized...

This section introduces you to Kafka architecture. By the end of this section, you will have a clear understanding of both the logical and physical architecture of Kafka. Let's see how Kafka components are organized logically.

Every message in Kafka topics is a collection of bytes. This collection is represented as an array. Producers are the applications that store information in Kafka queues. They send messages to Kafka topics that can store all types of messages. Every topic is further differentiated into partitions. Each partition stores messages in the sequence in which they arrive. There are two major operations that producers/consumers can perform in Kafka. Producers append to the end of the write-ahead log files. Consumers fetch messages from these log files belonging to a given topic partition. Physically, each topic is spread over different...

If you are into software development and services, I am sure you will have heard terms such as database, tables, records, and so on. In a database, we have multiple tables; let's say, Items, Price, Sales, Inventory, Purchase, and many more. Each table contains data of a specific category. There will be two parts in the application: one will be inserting records into these tables and the other will be reading records from these tables. Here, tables are the topics in Kafka, applications that are inserting data into tables are producers, and applications that are reading data are consumers.

In a messaging system, messages need to be stored somewhere. In Kafka, we store messages into topics. Each topic belongs to a category, which means that you may have one topic storing item information and another may store sales information. A producer who wants to send a message...

Suppose that we have in our possession a purchase table and we want to read records for an item from the purchase table that belongs to a certain category, say, electronics. In the normal course of events, we will simply filter out other records, but what if we partition our table in such a way that we will be able to read the records of our choice quickly?

This is exactly what happens when topics are broken into partitions known as units of parallelism in Kafka. This means that the greater the number of partitions, the more throughput. This does not mean that we should choose a huge number of partitions. We will talk about the pros and cons of increasing the number of partitions further.

While creating topics, you can always mention the number of partitions that you require for a topic. Each of the messages will be appended to partitions and each message is...

Replication is one of the most important factors in achieving reliability for Kafka systems. Replicas of message logs for each topic partition are maintained across different servers in a Kafka cluster. This can be configured for each topic separately. What it essentially means is that for one topic, you can have the replication factor as 3 and for another, you can use 5. All the reads and writes happen through the leader; if the leader fails, one of the followers will be elected as leader.

Generally, followers keep a copy of the leader's log, which means that the leader does not make the message as committed until it receives acknowledgment from all the followers. There are different ways that the log replication algorithm has been implemented; it should ensure that, if leader tells the producer that the message is committed, it must be available...

In Kafka, the producer is responsible for sending data to the partition of the topic for which it is producing data.

The hash value is generally calculated by the message key that we provide when writing the message to a Kafka topic. The message with a null key will be distributed in a round-robin fashion across partitions to ensure even distribution of messages. In Kafka, each partition has a leader and each read write request goes through the leader only. So a request to write messages to a partition of a topic will go through the leader broker. The producer waits for an acknowledgement of messages depending...

The consumer is any one who subscribes for topics in Kafka. Each consumer belongs to a consumer group and some consumer groups contains multiple consumers. Consumers are an interesting part of Kafka and we will cover them in detail.

So, it's clear that groups play an important role; in Kafka's initial version, Zookeeper was used for group management, but in the latest version, Kafka has its own group protocol built in. One of the brokers will act...

We have already talked a lot about Zookeeper in the previous sections. Zookeeper plays a very important role in Kafka architecture and it is very important for you to understand how it records the Kafka cluster state. Therefore, we are dedicating a separate section to the role of Zookeeper in the Kafka cluster. Kafka cannot work without Zookeeper. Kafka uses Zookeeper for the following functions:

• Choosing a controller: The controller is one of the brokers responsible for partition management with respect to leader election, topic creation, partition creation, and replica management. When a node or server shuts down, Kafka controllers elect partition leaders from followers. Kafka uses Zookeeper's metadata information to elect a controller. Zookeeper ensures that a new controller is elected in case the current controller crashes.
• Brokers metadata: Zookeeper...

We have come to the end of this chapter, and by now you should have a basic understanding of the Kafka messaging system. An important aspect of mastering any system is that you should understand the system end to end at a high level first. This will put you in a better position when you understand individual components of the system in detail. You can always establish the logical connection with end-to-end system understanding and understand why individual components are designed in a particular way. In this chapter, our goal was the same.

We started by discovering why Kafka was built in the first place. We have put forward problems in LinkedIn systems that led to the creation of Kafka. That section will give you a very clear understanding of the types of problem that Kafka can solve.

We further covered Kafka's logical and system architecture. Putting Kafka architecture...