Preprocessing data within Elasticsearch with ingest nodes

We gave you a brief overview about ingest nodes under the Node types in Elasticsearch section of Chapter 8, ElasticSearch Administration. In this section, we are going to cover ingest node functionalities in detail.

Ingest nodes, which are introduced in Elasticsearch 5.0, help in preprocessing the data and enriching them before they are actually indexed. This helps a lot in scenarios where you have to use a custom parser or Logstash for processing documents and enriching them, before sending to Elasticsearch. Now you can do all those things within Elasticsearch itself. This preprocessing is achieved with the help of defining a pipeline and a series of one or more processors. Each processor transforms the document in some way. For example, you can add a new field with a custom value or remove a field completely.

Working with ingest pipeline

An ingest pipeline has the following structure:

{ 
  "description" : "......