Configuring Kafka topics

The Kafka cluster is running, but the magic inside a broker is the queues, that is, the topics. This recipe shows the second step: how to create Kafka topics.

Getting ready

At this point, you need to:

• Have installed Kafka
• Have Zookeeper up and running
• Have a Kafka server up and running
• Go to the Kafka installation directory (/usr/local/kafka/ for Mac users and /opt/kafka/ for Linux users):

cd /usr/local/kafka

How to do it...

Recall that almost all modern projects have two ways to do things: through the command line and through code. Yes, believe or not, the Kafka brokers' creation can be done through code in almost all the modern programming languages; the previous recipe showed just the command-line method. In later chapters, the process to achieve it programmatically is explained.

The same goes for the topics. They can be created through the command line and through code. In this recipe, we will show it through the command line. Kafka has built-in utilities to create brokers (as already shown) and topics. From the Kafka installation directory, type the following command:

> ./bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic humbleTopic

The output should be:

Created topic "humbleTopic".

How it works...

Here, the kafka-topics.sh command is used. With the --create parameter, it is specified that we want to create a new topic. The --topic parameter set the name of the topic; in this case, humbleTopic.

The --replication-factor parameter is very important; it specifies how many servers of the cluster the topic is going to be replicated in (we mean, running). One broker can run just one replica. Obviously, if we specify a number greater than the number of running servers on the cluster, it is an error (don't be shy and try it in your environment), like this:

Error while executing topic command : replication factor: 3 larger than available brokers: 1
[2017-02-28 07:13:31,350] ERROR org.apache.kafka.common.errors.InvalidReplicationFactorException: replication factor: 3 larger than available brokers: 1
   (kafka.admin.TopicCommand$)

The --partitions parameter, as its name implies, says how many partitions our topic will have. The number of partitions determines the parallelism that can be achieved on the consumer's side. This parameter is fundamental when doing fine tuning on the cluster.

Finally, the --zoookeeper parameter indicates where the Zookeeper cluster is running.

When a topic is created, the output in the broker log is something like this:

[2017-02-28 07:05:53,910] INFO [ReplicaFetcherManager on broker 1] Removed fetcher for partitions humbleTopic-0 (kafka.server.ReplicaFetcherManager)
[2017-02-28 07:05:53,950] INFO Completed load of log humbleTopic-0 with 1 log segments and log end offset 0 in 21 ms (kafka.log.Log)

This message says that a new topic has been born in that broker.

There's more…

Yes, there are more parameters than --create. To check whether a topic has been successfully created, run the kafka-topics command with the --list parameter:

> ./bin/kafka-topics.sh --list --ZooKeeper localhost:2181 humbleTopic

This parameter returns the list of all the existent topics in the Kafka cluster.

To get the details of a particular topic, run the kafka-topics command with the --describe parameter:

> ./bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic humbleTopic

The command output is:

Topic:humbleTopic     PartitionCount:1      ReplicationFactor:1   Configs:
Topic: humbleTopic    Partition: 0  Leader: 1     Replicas: 1   Isr: 1

The explanation of the output is:

• PartitionCount: Number of partitions existing on this topic.
• ReplicationFactor: Number of replicas existing on this topic.
• Leader: Node responsible for the reading and writing operations of a given partition.
• Replicas: List of brokers replicating the Kafka data. Some of these might even be dead.
• ISR: List of nodes that are currently in-sync replicas.

To create a topic with multiple replicas, we need to increase the replication factor as follows:

> ./bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 2 --partitions 1 --topic replicatedTopic

The output is as follows:

Created topic "replicatedTopic".

Call the kafka-topics command with the --describe parameter to check the topic details:

> ./bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic replicatedTopic
Topic:replicatedTopic PartitionCount:1      ReplicationFactor:2   Configs:
Topic: replicatedTopic       Partition: 0  Leader: 1     Replicas: 1,2 Isr: 1,2

As Replicas and ISR (in-sync replicas) are the same lists, all the nodes are in-sync.

Try to play with all these commands; try to create replicated topics on dead servers and see the output. Also, create topics on running servers and then kill them to see the results.

As mentioned before, all the commands executed through the command line can be executed programmatically.