Pushing data from Kafka to Elastic

As mentioned, Elasticsearch is a distributed, full-text search engine that supports a RESTful web interface and schema-free JSON documents. Elasticsearch was built with distributed searches in mind. There are several ways to push data into Elasticsearch. In this recipe, the plugin that enables data pushing from Kafka to Elasticsearch is analyzed.

Getting ready

For this recipe, a Kafka cluster must be up and running and a confluent platform is needed. To install Elasticsearch follow the instructions on this page: https://www.elastic.co/guide/en/elasticsearch/reference/current/_installation.html.

How to do it...

The Elasticsearch connector is needed. Before starting the connector, the configuration is in

etc/kafka-connect-elasticsearch/quickstart-elasticsearch.properties.

This must be properly set to the Elasticsearch configuration: connection.url points to the correct HTTP address.

To start the Elasticsearch connector, use the following command:

$ ./bin/connect...