Moving data between Kafka nodes with Flume

Apache Flume is a reliable, highly available, distributed service for collecting, aggregating, and moving large amounts of data logs into data storage solutions. The data storage destination might be HDFS, Kafka, Hive, or any of the various sinks that Flume supports.

Apache Flume can also be used to transfer data between Kafka nodes. The following recipe shows how to do that.

Getting ready

For this recipe, it is necessary to have two different Kafka brokers up and running, one to publish data (source-topic) and the other (target-topic) to receive data.

The installation of Apache Flume is also required. Follow the instructions on this page: https://flume.apache.org/download.html.

How to do it...

1. In the conf folder, create a Flume configuration file called flume.conf with this content:

flume1.sources = kafka-source-1 
flume1.channels = mem-channel-1 
flume1.sinks = kafka-sink-1 
 
flume1.sources.kafka-source-1.type=org.apache.flume.source.kafka.KafkaSource...