Supporting indices

For Elastic's ML to function, there are several supporting indices that exist and serve specific purposes. We will look at the following indices and describe their roles:

• .ml-state
• .ml-notifications
• .ml-anomalies-*

.ml-state

The .ml-state index is the place where ML keeps the internal information about the statistical models that have been learned for a specific dataset, plus additional logistical information. This index is not meant to be understandable by a user—it is the backend algorithms of ML that will read and write entries in this index.

Information in the .ml-state index is compressed and is a small fraction of the size of the raw data that the ML jobs are analyzing.

.ml-notifications

The .ml-notifications index stores the audit messages for ML that appear in the Job messages section of the Job Management page of the UI:

Audit messages for a particular job in the ML UI

These messages convey the basic information about the job's creation and activity. Additionally, basic operational errors can be found here. Detailed information about the execution of ML jobs, however, is found in the elasticsearch.log file.

.ml-anomalies-*

The .ml-anomalies-* indices contain the detailed results of ML jobs. There is a single .ml-anomalies-shared index that can contain information from multiple jobs (keyed with the job_id field). If the user chooses to Use a dedicated index in the user interface when creating a job (or sets the results_index_name when using the API), then a dedicated results index for that job will be created.

These indices are instrumental in leveraging the output of the ML algorithms. All information displayed in the ML UI will be driven from this result data. Additionally, proactive alerting on anomalies will be accomplished by having watches configured against these indices. More information on this will be presented in Chapter 6, Alerting on ML Analysis.