Setting up a data stream manually

Data streams are a new feature introduced in Elasticsearch 7.9 and solidified in version 8. They allow for more efficient management and reduced overhead of time series data. As you learned in the previous recipes of this chapter, both Elastic Agent and Beats create timestamped data automatically in data streams. What about custom data sources where we cannot easily install Elastic Agent or Beats as data shippers? By following the steps in this recipe’s How to do it… section, you’ll be able to create a data stream and start ingesting and analyzing time-based data manually in Elasticsearch.

Dataset

In this and the following recipe, we will use a free dataset provided by European public sector datasets (https://data.europa.eu/) – specifically, a real-time traffic status dataset for the city of Rennes, one of the biggest cities in France (https://data.europa.eu/data/datasets/5caaf5ee9ce2e75d0c8c381a?locale=en).

Dataset...