Expanding clusters

Adding nodes to an existing cluster is not the same as building a new Kafka cluster. Adding nodes to an existing cluster is easy. We do this by assigning them a unique broker ID, but they are not going to receive data automatically. A cluster reconfiguration is needed to indicate which partition replicas go where. Then, the partitions will move to the newly added nodes. This recipe shows how to do that.

Getting ready

For this recipe, Kafka must be installed, ZooKeeper should be running, and the broker should be running with some topics created on it.

How to do it...

1. This recipe moves all partitions for existing topics: topic_1 and topic_2. The newly generated brokers are broker_7 and broker_8 (suppose that brokers 1 to 6 already exist). After finishing the movement, all partitions for topic_1 and topic_2 will exist only in broker_7 and broker_8.
2. The tool only accepts JSON files as input; let's create the JSON file as follows:

$ cat to_reassign.json 
 
{"topics": [{"topic": ...