Configuring Logstash input

In this section, we'll configure Logstash to read data from access logs located on Tomcat, and index it in Elasticsearch, making filters and tokenization of terms in logs as per the grok pattern.

Grok pattern for access logs

As we already saw, some of the commonly used grok patterns are already included with the Logstash installation. Check out the list of Logstash grok patterns on GitHub at https://github.com/logstash-plugins/logstash-patterns-core/tree/master/patterns.

There is already a grok pattern for the Common Apache log format in the Logstash installation as follows:

COMMONAPACHELOG %{IPORHOST:clientip} %{USER:ident} %{USER:auth} \[%{HTTPDATE:timestamp}\] "(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})" %{NUMBER:response} (?:%{NUMBER:bytes}|-)

We can directly use COMMONAPACHELOG as a matching pattern for our incoming messages to Logstash as follows:

input{
file{
path =>"/var/lib/tomcat7/logs/localhost_access_logs.txt...