Logstash architecture

The Logstash event processing pipeline has three stages, they are: Inputs, Filters and Outputs. A Logstash pipeline has two required elements; input, output, and, optionally, filters:

Inputs create events, Filters modify the input events, and Outputs ship them to the destination. Inputs and outputs support codecs which enable you to encode or decode the data as and when it enters or exits the pipeline without having to use a separate filter.

Logstash uses in-memory bounded queues between pipeline stages by default (Input to Filter and Filter to Output) to buffer events. If Logstash terminates unsafely, any events that are stored in memory will be lost. To prevent data loss, you can enable Logstash to persist in-flight events to the disk by making use of persistent queues. 

Note

Persistent queues can be enabled by setting the property queue.type: persisted in the logstash.yml file found under the LOGSTASH_HOME/config folder. logstash.yml is a configuration file containing...