Manipulating incoming data with ingest pipelines

Elasticsearch is a "schema on write" data store. Once a document has been indexed into Elasticsearch, the field names and values that have been indexed cannot be changed unless the document is reindexed. Therefore, documents must be parsed, transformed, and cleansed before ingestion.

Runtime fields can be used to compute or evaluate the value of a field at query time. Runtime fields can be used to manipulate and transform field values when searching for data, but they can be costly and time-consuming to run across large volumes of search requests. The intended use of runtime fields is to apply temporary or one-off changes to data, rather than on every search request.

Ingest pipelines on Elasticsearch offer lightweight and convenient data transformation and manipulation functionality for when an ETL tool such as Logstash is not used. As ingest pipelines run on Elasticsearch nodes, they can scale easily as part of the...