Collecting logs
The first step in any log management application is to collect the logs in a common storage repository for analyzing them later and archiving them for the records. That effort involves shipping the log files from machines and services where they are available to the common storage repository.
The following diagram provides the workflow of collecting and processing the logs and rendering the aggregated information to end users. The aggregated information could be published as metrics, which could be used for setting up monitors. That is the same as using metrics to set up monitors in a conventional monitoring application:
Figure 13.1 – Log management workflow
In a modern production infrastructure, the logs could be generated by a variety of sources, and typical sources include the following:
- Public cloud services: Public cloud services such as AWS S3 and RDS are very popular, especially if the production infrastructure is...
