Creating a Logstash pipeline

Creating a Logstash pipeline involves defining a series of configurations that specify how data should be collected, parsed, filtered, and where it should be sent for further analysis or storage. This process enables you to harmonize data from various sources, making it ready for visualization, search, and analysis through Elasticsearch and Kibana, as well as third-party destinations.

In this recipe, we will walk you through the steps to create a Logstash pipeline, from configuring input sources to defining filters, specifying output destinations, and running the pipeline. Our example is based on Rennes Traffic Data, which we introduced in Chapter 4.

Getting ready

You will need to have completed the previous Installing self-managed Logstash recipe and the Setting up time series data stream (TSDS) manually recipe in Chapter 4 as we are going to reuse the objects that we created in this recipe such as the index life cycle policy, mapping, setting...