Testing the Kafka cluster

Kafka comes with scripts to allow you to perform some basic functions from the command line. To test the cluster, you can create a topic, create a producer, send some messages, and then create a consumer to read them. If the consumer can read them, your cluster is running.

To create a topic, run the following command from your kafka_1 directory:

bin/kafka-topics.sh --create --zookeeper localhost:2181,localhost:2182,localhost:2183 --replication-factor 2 --partitions 1 --topic dataengineering

The preceding command runs the kafka-topics script with the create flag. It then specifies the ZooKeeper cluster IP addresses and the topic. If the topic was created, the terminal will have printed the following line:

created topic dataengineering

You can verify this by listing all the topics in the Kafka cluster using the same script, but with the list flag:

bin/kafka-topics.sh –list --zookeeper localhost:2181,localhost:2182,localhost:2183

The...