The birth of Elastic Stack

All the jobs of reading data were once done using Logstash, but that's resource consuming. Since Logstash runs on JVM, it consumes a good amount of memory. The community realized the need for improvement and to make the pipelining process resource friendly and lightweight. In 2015, Packetbeat was born, a project which was an effort to make a network packet analyzer that could read from different protocols, parse the data, and ship to Elasticsearch. Being lightweight in nature did the trick and a new concept of Beats was formed. Beats are written in Go programming language. The project evolved, and now ELK stack was no more just Elasticsearch, Logstash, and Kibana;  Beats also became a significant component.

The pipeline now looked as follows:

Beat

A Beat reads data, parses it, and can ship it to either Elasticsearch or Logstash. The difference is that they are lightweight, serve a specific purpose, and are installed as agents. There are a few Beats available such as Metricbeat, Filebeat, Packetbeat, and so on, which are supported and provided by the Elastic Team and a good number of Beats are already written by the community. If you have a specific requirement, you can write your own Beat using the libbeat library.

In simple words, Beats can be treated as very lightweight agents to ship data to either Logstash or Elasticsearch, offering you an infrastructure using the libbeat library to create your own Beats.

We will be learning more about Beats in Chapter 5, Using Beats and Chapter 7, Customizing Elastic Stack.

Together Elasticsearch, Logstash, Kibana, and Beats became Elastic Stack, formally known as ELK Stack. Elastic Stack did not just add Beats to its team; they will be using the same version always. The starting version of the Elastic Stack will be 5.0.0 and the same version will apply to all the components.

This version and release method is not only for Elastic Stack, but for other tools of the Elastic family as well. Due to there being so many tools, there was a problem of unification, wherein each tool had their own version, and every version was not compatible with each other, hence leading to a problem. To solve this, all of the tools will now be built, tested, and released together.

All of these components play a significant role in creating a pipeline. While Beats and Logstash are used to collect the data, parse it, and ship it, Elasticsearch creates indices, which is finally used by Kibana to make visualizations. While Elastic Stack helps with a pipeline, other tools add security, notifications, monitoring, and other such capabilities to the setup.