Ingesting data from Kafka to Storm

Apache Storm is a real-time, distributed stream-processing system. Storm simplifies real-time data processing, Kafka can work as the source for this data streaming.

Getting ready

Have a Kafka cluster up and running. To install Apache Storm follow the instructions on this page: http://storm.apache.org/downloads.html.

How to do it...

Storm has a built-in KafkaSpout to easily ingest data from Kafka to the Storm topology:

1. The first step is to create the ZkHosts object with the ZooKeeper address in host:port format:

BrokerHosts hosts = new ZkHosts("127.0.0.1:2181"); 

2. Next, create the SpoutConfig object that contains the parameters needed for KafkaSpout:

SpoutConfig kafkaConf = new SpoutConfig(hosts,"source-topic", "/brokers", "kafkaStormTest");

3. Then, declare the scheme for the KafkaSpout config:

kafkaConf.scheme = new SchemeAsMultiScheme(new StringScheme()); 

4. Using this scheme, create a KafkaSpout object:

KafkaSpout kafkaSpout = new KafkaSpout(kafkaConf);

5. Build that topology...