Ingesting data in OpenSearch

In this section, we will work around the system described in Figure 12.1. There, you can see data-ingesting components on the left of OpenSearch. On the other side, OpenSearch feeds Grafana dashboards:

Figure 12.1 – System architecture

You can ingest data in OpenSearch by sending PUT or POST HTTP requests. So, you can send data to OpenSearch from a script, using curl or some similar command.

In this section, you will learn how to implement a data ingestion pipeline for use in IoT systems. In this case, we will use Fluent Bit. Let’s see how to do it.

Fluent Bit

Fluent Bit is an open source log processor. It lets you gather data, process it, and send it to other systems, such as OpenSearch.

You can reach this project on Github at the following link: https://github.com/fluent/fluent-bit.

You can install Fluent Bit on several Linux distributions, several versions of Windows, and macOS. To see detailed instructions...