The value of running machine learning on Elasticsearch
Elasticsearch is a powerful tool when it comes to storing, searching, and aggregating large volumes of data. Dashboards and visualizations help with user-driven interrogation and exploration of data, while tools such as Watcher and Kibana alerting allow users to take automatic action when data changes in a predefined or expected manner.
However, a lot of data sources can often represent trends or insights that are hard to capture as a predefined rule or query. Consider the following example:
- A logging platform collects application logs (using an agent) from about 5,000 endpoints across an environment.
- The application generates a log line for every transaction executed as soon as the transaction completes.
- After a software patch, a small subset of the endpoints can intermittently and temporarily fail to write logs successfully. The machine doesn't entirely fail as the failure is intermittent in nature. ...