Logstash Configuration for Parsing Logs

In this section, we will explore how to use Logstash to parse a file that contains different types of logs. It will utilize the knowledge gained in this chapter, which will be put into action. We will use custom created grok patterns to parse the data, as per our requirements.

Let's have a look at the data.

The log file contains millions of records with a combination of Tomcat logs and Catalina logs. The log file also contains application exceptions, errors, and stack trace messages. The log file contains log events of various log levels, such as INFO, WARN, ERROR, DEBUG, and FATAL.

Sample Catalina logs

Have a look at the following logs:

    Mar 10, 2016 10:04:37 PM org.apache.catalina.startup.Catalina load INFO: Initialization processed in 433 ms
    Mar 10, 2016 10:04:37 PM org.apache.catalina.core.StandardService startInternal INFO: Starting service Catalina

Sample Tomcat logs

Have a look at the following logs:

    2016-03-10 22:04:40,892  INFO localhost...