Every application and device logs in its own special way, so each format needs its own expert. Also, it is difficult to search across because of different formats.

Let's take a look at some of the common log formats. An interesting thing to observe will be the way different logs represent different timestamp formats, different ways to represent INFO, ERROR, and so on, and the order of these components with logs. It's difficult to figure out just by seeing logs what is present at what location. This is where tools such as Logstash help.

May 24, 2015 3:56:26 PM org.apache.catalina.startup.HostConfig deployWAR
INFO: Deployment of web application archive \soft\apache-tomcat-7.0.62\webapps\sample.war has finished in 253 ms

127.0.0.1 - - [24/May/2015:15:54:59 +0530] "GET /favicon.ico HTTP/1.1" 200 21630

2012-05-02 17:42:15 172.24.255.255 - 172.20.255.255 80 GET /images/favicon.ico - 200 Mozilla/4.0+(compatible;MSIE+5.5;+Windows+2000+Server)

Not only log formats, but timestamp formats are also different among different types of applications, different types of events generated across multiple devices, and so on. Different types of time formats across different components of your system also make it difficult to correlate events occurring across multiple systems at the same time:

People interested in getting the required business-centric information out of logs generally don't have access to the logs or may not have the technical expertise to figure out the appropriate information in the quickest possible way, which can make analysis slower, and sometimes, impossible too.

Elasticsearch is a distributed open source search engine based on Apache Lucene, and released under an Apache 2.0 license (which means that it can be downloaded, used, and modified free of charge). It provides horizontal scalability, reliability, and multitenant capability for real-time search. Elasticsearch features are available through JSON over a RESTful API. The searching capabilities are backed by a schema-less Apache Lucene Engine, which allows it to dynamically index data without knowing the structure beforehand. Elasticsearch is able to achieve fast search responses because it uses indexing to search over the texts.

Elasticsearch is used by many big companies, such as GitHub, SoundCloud, FourSquare, Netflix, and many others. Some of the use cases are as follows:

Some of the key features of Elasticsearch are:

Logstash is a data pipeline that helps collect, parse, and analyze a large variety of structured and unstructured data and events generated across various systems. It provides plugins to connect to various types of input sources and platforms, and is designed to efficiently process logs, events, and unstructured data sources for distribution into a variety of outputs with the use of its output plugins, namely file, stdout (as output on console running Logstash), or Elasticsearch.

It has the following key features:

Kibana is an open source Apache 2.0 licensed data visualization platform that helps in visualizing any kind of structured and unstructured data stored in Elasticsearch indexes. Kibana is entirely written in HTML and JavaScript. It uses the powerful search and indexing capabilities of Elasticsearch exposed through its RESTful API to display powerful graphics for the end users. From basic business intelligence to real-time debugging, Kibana plays its role through exposing data through beautiful histograms, geomaps, pie charts, graphs, tables, and so on.

Kibana makes it easy to understand large volumes of data. Its simple browser-based interface enables you to quickly create and share dynamic dashboards that display changes to Elasticsearch queries in real time.

Some of the key features of Kibana are as follows:

When installing Elasticsearch during production, you can use the method described below, or the Debian or RPM packages provided on the download page.

curl –O https://download.elastic.co/elasticsearch/elasticsearch/elasticsearch-1.5.2.tar.gz 

sudo apt-get install curl

Then, unpack the zip file on your local filesystem:

tar -zxvf elasticsearch-1.5.2.tar.gz

And then, go to the installation directory:

cd  elasticsearch-1.5.2

In order to run Elasticsearch, execute the following command:

$ bin/elasticsearch

Add the -d flag to run it in the background as a daemon process.

We can test it by running the following command in another terminal window:

curl 'http://localhost:9200/?pretty'

This shows you an output similar to this:

{
  "status" : 200,
  "name" : "Master",
  "cluster_name" : "elasticsearch",
  "version" : {
    "number" : "1.5.2",
    "build_hash" : "c88f77ffc81301dfa9dfd81ca2232f09588bd512",
    "build_timestamp" : "2015-05-13T13:05:36Z",
    "build_snapshot" : false,
    "lucene_version" : "4.10.3"
  },
  "tagline" : "You Know, for Search"
}

We can shut down Elasticsearch through the API as follows:

curl -XPOST 'http://localhost:9200/_shutdown'

Elasticsearch configuration files are under the config folder in the Elasticsearch installation directory. The config folder has two files, namely elasticsearch.yml and logging.yml. The former will be used to specify configuration properties of different Elasticsearch modules, such as network address, paths, and so on, while the latter will specify logging-related configurations.

The configuration file is in the YAML format and the following sections are some of the parameters that can be configured.

network :
    host : 127.0.0.1

path:
  logs: /var/log/elasticsearch
  data: /var/data/elasticsearch

cluster:
  name: <NAME OF YOUR CLUSTER>

node:
  name: <NAME OF YOUR NODE>

Elasticsearch has a variety of plugins that ease the task of managing indexes, cluster, and so on. Some of the mostly used ones are the Kopf plugin, Marvel, Sense, Shield, and so on, which will be covered in the subsequent chapters. Let's take a look at the Kopf plugin here.

Kopf is a simple web administration tool for Elasticsearch that is written in JavaScript, AngularJS, jQuery and Twitter bootstrap. It offers an easy way of performing common tasks on an Elasticsearch cluster. Not every single API is covered by this plugin, but it does offer a REST client, which allows you to explore the full potential of the Elasticsearch API.

In order to install the elasticsearch-kopf plugin, execute the following command from the Elasticsearch installation directory:

bin/plugin -install lmenezes/elasticsearch-kopf

Now, go to this address to see the interface: http://localhost:9200/_plugin/kopf/.

You can see a page similar to this, which shows Elasticsearch nodes, shards, a number of documents, size, and also enables querying the documents indexed.

Elasticsearch Kopf UI

First, download the latest Logstash TAR file from the download page.

curl –O http://download.elastic.co/logstash/logstash/logstash-1.5.0.tar.gz

Then, unpack the GZIP file on your local filesystem:

tar -zxvf logstash-1.5.0.tar.gz

Now, you can run Logstash with a basic configuration.

Run Logstash using -e flag, followed by the configuration of standard input and output:

cd logstash-1.5.0
bin/logstash -e 'input { stdin { } } output { stdout {} }'

Now, when we type something in the command prompt, we will see its output in Logstash as follows:

hello logstash
2015-05-15T03:34:30.111Z 0.0.0.0 hello logstash

Here, we are running Logstash with the stdin input and the stdout output as this configuration prints whatever you type in a structured format as the output. The -e flag allows you to quickly test the configuration from the command line.

Now, let's try the codec setting for output for a pretty formatted output. Exit from the running Logstash by issuing a Ctrl + C command, and then we need to restart Logstash with the following command:

bin/logstash -e 'input { stdin { } } output { stdout { codec => rubydebug } }'

Now, enter some more test input:

Hello PacktPub

{
  "message" => " Hello PacktPub",
  "@timestamp" => "2015-05-20T23:48:05.335Z",
  "@version" => "1",
  "host" => "packtpub"
}

The output that you see is the most common output that we generally see from Logstash:

Logstash can be easily configured to read from a log file as input.

For example, to read Apache logs from a file and output to a standard output console, the following configuration will be helpful:

input {
  file {
    type => "apache"
    path => "/user/packtpub/intro-to-elk/elk.log"
  }
}
output {
  stdout {
    codec => rubydebug
  }
}

Logstash can be configured to output all inputs to an Elasticsearch instance. This is the most common scenario in an ELK platform:

bin/logstash -e 'input { stdin { } } output { elasticsearch { host = localhost } }'

Then type 'you know, for logs

You will be able to see indexes in Elasticsearch through http://localhost:9200/_search.

Logstash configuration files are in the JSON format. A Logstash config file has a separate section for each type of plugin that you want to add to the event processing pipeline. For example:

# This is a comment. You should use comments to describe
# parts of your configuration.
input {
  ...
}

filter {
  ...
}

output {
  ...
}

Each section contains the configuration options for one or more plugins. If you specify multiple filters, they are applied in the order of their appearance in the configuration file.

When you run logstash, you use the -flag to read configurations from a configuration file or even from a folder containing multiple configuration files for each type of plugin—input, filter, and output:

bin/logstash –f ../conf/logstash.conf

bin/logstash –configtest ../conf/logstash.conf

Logstash runs on JVM and consumes a hefty amount of resources to do so. Logstash, at times, has significant memory consumption. Obviously, this could be a great challenge when you want to send logs from a small machine without harming application performance.

In order to save resources, you can use the Logstash forwarder (previously known as Lumberjack). The forwarder uses Lumberjack's protocol, enabling you to securely ship compressed logs, thus reducing resource consumption and bandwidth. The sole input is file/s, while the output can be directed to multiple destinations.

Other options do exist as well, to send logs. You can use rsyslog on Linux machines, and there are other agents for Windows machines, such as nxlog and syslog-ng. There is another lightweight tool to ship logs called Log-Courier (https://github.com/driskell/log-courier), which is an enhanced fork of the Logstash forwarder with some improvements.

Download the latest Logstash forwarder release from the download page.

Prepare a configuration file that contains input plugin details and ssl certificate details to establish a secure communication between your forwarder and indexer servers, and run it using the following command:

Logstash forwarder -config Logstash forwarder.conf

And in Logstash, we can use the Lumberjack plugin to get data from the forwarder:

input {
  lumberjack {
    # The port to listen on
    port => 12345

    # The paths to your ssl cert and key
    ssl_certificate => "path/to/ssl.crt"
    ssl_key => "path/to/ssl.key"

    # Set the type of log.
    type => "log type"
  }

Some of the most popular Logstash plugins are:

Before we can install and run Kibana, it has certain prerequisites:

Download the latest Kibana release from the download page.

curl –O https://download.elastic.co/kibana/kibana/kibana-4.0.2-linux-x64.tar.gz

Then, unpack kibana-4.0.2-linux-x64.tar.gz on your local file system and create a soft link to use a short name.

tar -zxvf kibana-4.0.2-linux-x64.tar.gz

ln -s kibana-4.0.2-linux-x64 kibana

Then, you can explore the kibana folder:

cd kibana

The Kibana configuration file is present in the config folder inside the kibana installation:

config/kibana.yml

Following are some of the important configurations for Kibana.

This controls which port to use.

port: 5601.

Property to set the host to bind the server is:

host: "localhost".

Set the elasticsearch_url to point at your Elasticsearch instance, which is localhost by default.

elasticsearch_url: http://localhost:9200

Start Kibana manually by issuing the following command:

bin/kibana

You can verify the running Kibana instance on port 5601 by placing the following URL in the browser:

http://localhost:5601

This should fire up the Kibana UI for you.

Kibana UI

As you saw in the preceding screenshot of the Kibana UI, the Kibana interface consists of four main components—Discover, Visualize, Dashboard, and Settings.

Discover

The Discover page helps to interactively explore the data matching the selected index pattern. This page allows submitting search queries, filtering the search results, and viewing document data. Also, it gives us the count of matching results and statistics related to a field. If the timestamp field is configured in the indexed data, it will also display, by default, a histogram showing distribution of documents over time.

Kibana Discover Page

Visualize

The Visualize page is used to create new visualizations based on different data sources—a new interactive search, a saved search, or an existing saved visualization. Kibana 4 allows you to create the following visualizations in a new visualization wizard:

• Area chart
• Data table
• Line chart
• Markdown widget
• Metric
• Pie chart
• Tile map
• Vertical bar chart

These visualizations can be saved, used individually, or can be used in dashboards.

Kibana Visualize Page