Search icon CANCEL
Subscription
0
Cart icon
Your Cart (0 item)
Close icon
You have no products in your basket yet
Save more on your purchases! discount-offer-chevron-icon
Savings automatically calculated. No voucher code required.
Arrow left icon
Explore Products
Best Sellers
New Releases
Books
Videos
Audiobooks
Learning Hub
Newsletter Hub
Free Learning
Arrow right icon
timer SALE ENDS IN
0 Days
:
00 Hours
:
00 Minutes
:
00 Seconds
Learning ELK Stack
Learning ELK Stack

Learning ELK Stack: Build mesmerizing visualizations, analytics, and logs from your data using Elasticsearch, Logstash, and Kibana

eBook
₱1799.99 ₱2000.99
Paperback
₱2500.99
Subscription
Free Trial
Renews at $19.99p/m

What do you get with eBook?

Product feature icon Instant access to your Digital eBook purchase
Product feature icon Download this book in EPUB and PDF formats
Product feature icon Access this title in our online reader with advanced features
Product feature icon DRM FREE - Read whenever, wherever and however you want
OR
Modal Close icon
Payment Processing...
tick Completed

Billing Address

Table of content icon View table of contents Preview book icon Preview Book

Learning ELK Stack

Chapter 2. Building Your First Data Pipeline with ELK

In the previous chapter, we got familiar with each component of ELK Stack—Elasticsearch, Logstash, and Kibana. We got the components installed and configured. In this chapter, we will build our first basic data pipeline using ELK Stack. This will help us understand how easy it is to get together the components of ELK Stack to build an end-to-end analytics pipeline.

While running the example in this chapter, we assume that you already installed Elasticsearch, Logstash, and Kibana as described in Chapter 1, Introduction to ELK Stack.

Input dataset

For our example, the dataset that we are going to use here is the daily Google (GOOG) Quotes price dataset over a 6 month period from July 1, 2014 to December 31, 2014. This is a good dataset to understand how we can quickly analyze simple datasets, such as these, with ELK.

Note

This dataset can be easily downloaded from the following source:

http://finance.yahoo.com/q/hp?s=GOOG

Data format for input dataset

The most significant fields of this dataset are Date, Open Price, Close Price, High Price, Volume, and Adjusted Price.

The following table shows some of the sample data from the dataset. The actual dataset is in the CSV format.

Date

Open

High

Low

Close

Volume

Adj Close

Dec 31, 2014

531.25

532.60

525.80

526.40

1,368,200

526.40

Dec 30, 2014

528.09

531.15

527.13

530.42

876,300

530.42

Dec 29, 2014

532.19

535.48

530.01

530.33

2,278,500

530.33

Dec 26, 2014

528.77

534.25

527.31

534.03

1,036,000

534.03

Dec 24, 2014

530.51

531.76

527.02

528.77...

Configuring Logstash input

As we already know, Logstash has a rich set of plugins for different types of inputs, outputs and filters, which can read, parse, and filter data as per our needs. We will utilize the file input plugin to read the source file.

A file input plugin streams events from the input file, and each event is assumed as a single line. It automatically detects file rotation and handles it. It maintains the location where it left reading, and will automatically detect the new data if configured correctly. It reads files in a similar manner:

tail -0f 

In general, a file input plugin configuration will look as follows:

input {
 
file {
    path => #String (path of the files) (required) 
    start_position => #String (optional, default "end")
    tags => #array (optional)
    type => #string (optional)
}

}
  • path: The path field is the only required field in file input plugin, which represents the path of the file from where input events have to be processed...

Filtering and processing input

Once we configure the input file, we need to filter the input based on our needs so that we can identify which fields we need, and process them as per the required analysis.

A filter plugin will perform the intermediary processing on the input event. We can apply the filter conditionally based on certain fields.

Since our input file is a CSV file, we will use the csv filter for the same. The csv filter takes an event field that contains CSV formatted data, parses it, and stores it as individual fields. It can also parse data with any separator other than commas. A typical csv filter is as follows:

filter {  
    csv {
        columns => #Array of column names.
        separator => #String ; default -","
    }
}

The attribute columns take the name of fields in our CSV file, which is optional. By default, the columns will be named as column 1, column 2, and so on.

The attribute separator defines what character is used to separate the different columns...

Putting data to Elasticsearch

Now that we have set up the data to be consumed by a CSV file into Logstash, followed by parsing and processing based on the data type needed, we now need to put the data in Elasticsearch so that we can index the different fields and consume them later via the Kibana interface.

We will use the output plugin of Logstash for an elasticsearch output.

A typical elasticsearch plugin configuration looks like this:

output {

  elasticsearch {

    action =>  # string (optional), default: "index"

    cluster =>  # string (optional)

    host =>  # string (optional)
   
    document_id =>  # string (optional), default: nil


    index =>  # string (optional), default: "logstash-%{+YYYY.MM.dd}"
    index_type =>  # string (optional)
    port =>  # string (optional)
    protocol =>  # string, one of ["node", "transport", "http"] (optional)
  }
}
  • action: This specifies what action to perform on incoming...

Visualizing with Kibana

Now when you verify that your data is indexed successfully in Elasticsearch, we can go ahead and look at the Kibana interface to get some useful analytics from the data.

Running Kibana

As described in the previous chapter, we will start the Kibana service from the Kibana installation directory.

$ bin/kibana

Now, let's see Kibana up and running similar to the following screenshot on the browser, by going to the following URL:

http://localhost:5601

Running Kibana

Kibana Discover page

As we already set up Kibana to take logstash-* indexes by default, it displays the indexed data as a histogram of counts, and the associated data as fields in the JSON format.

First of all, we need to set the date filter to filter based on our date range so that we can build our analysis on the same. Since we took data from July 1, 2014 to December 31, 2014, we will configure our date filter for the same.

Clicking on the Time Filter icon at the extreme top-right corner, we can set an Absolute Time Filter...

Input dataset


For our example, the dataset that we are going to use here is the daily Google (GOOG) Quotes price dataset over a 6 month period from July 1, 2014 to December 31, 2014. This is a good dataset to understand how we can quickly analyze simple datasets, such as these, with ELK.

Note

This dataset can be easily downloaded from the following source:

http://finance.yahoo.com/q/hp?s=GOOG

Data format for input dataset

The most significant fields of this dataset are Date, Open Price, Close Price, High Price, Volume, and Adjusted Price.

The following table shows some of the sample data from the dataset. The actual dataset is in the CSV format.

Date

Open

High

Low

Close

Volume

Adj Close

Dec 31, 2014

531.25

532.60

525.80

526.40

1,368,200

526.40

Dec 30, 2014

528.09

531.15

527.13

530.42

876,300

530.42

Dec 29, 2014

532.19

535.48

530.01

530.33

2,278,500

530.33

Dec 26, 2014

528.77

534.25

527.31

534.03

1,036,000

534.03

Dec 24, 2014

530.51

531.76

527.02

528.77

...
Left arrow icon Right arrow icon

Description

The ELK stack—Elasticsearch, Logstash, and Kibana, is a powerful combination of open source tools. Elasticsearch is for deep search and data analytics. Logstash is for centralized logging, log enrichment, and parsing. Kibana is for powerful and beautiful data visualizations. In short, the Elasticsearch ELK stack makes searching and analyzing data easier than ever before. This book will introduce you to the ELK (Elasticsearch, Logstash, and Kibana) stack, starting by showing you how to set up the stack by installing the tools, and basic configuration. You’ll move on to building a basic data pipeline using the ELK stack. Next, you’ll explore the key features of Logstash and its role in the ELK stack, including creating Logstash plugins, which will enable you to use your own customized plugins. The importance of Elasticsearch and Kibana in the ELK stack is also covered, along with various types of advanced data analysis, and a variety of charts, tables ,and maps. Finally, by the end of the book you will be able to develop full-fledged data pipeline using the ELK stack and have a solid understanding of the role of each of the components.

Who is this book for?

If you are a developer or DevOps engineer interested in building a system that provides amazing insights and business metrics out of data sources, of various formats and types, using the open source technology stack that ELK provides, then this book is for you. Basic knowledge of Unix or any programming language will be helpful to make the most out of this book.

What you will learn

  • Install, configure, and run Elasticsearch, Logstash, and Kibana
  • Understand the need for log analytics and the current challenges in log analysis
  • Build your own data pipeline using the ELK stack
  • Familiarize yourself with the key features of Logstash and the variety of input, filter, and output plugins it provides
  • Build your own custom Logstash plugin
  • Create actionable insights using charts, histograms, and quick search features in Kibana4
  • Understand the role of Elasticsearch in the ELK stack

Product Details

Country selected
Publication date, Length, Edition, Language, ISBN-13
Publication date : Nov 26, 2015
Length: 206 pages
Edition : 1st
Language : English
ISBN-13 : 9781785886706
Vendor :
Elastic
Category :

What do you get with eBook?

Product feature icon Instant access to your Digital eBook purchase
Product feature icon Download this book in EPUB and PDF formats
Product feature icon Access this title in our online reader with advanced features
Product feature icon DRM FREE - Read whenever, wherever and however you want
OR
Modal Close icon
Payment Processing...
tick Completed

Billing Address

Product Details

Publication date : Nov 26, 2015
Length: 206 pages
Edition : 1st
Language : English
ISBN-13 : 9781785886706
Vendor :
Elastic
Category :

Packt Subscriptions

See our plans and pricing
Modal Close icon
$19.99 billed monthly
Feature tick icon Unlimited access to Packt's library of 7,000+ practical books and videos
Feature tick icon Constantly refreshed with 50+ new titles a month
Feature tick icon Exclusive Early access to books as they're written
Feature tick icon Solve problems while you work with advanced search and reference features
Feature tick icon Offline reading on the mobile app
Feature tick icon Simple pricing, no contract
$199.99 billed annually
Feature tick icon Unlimited access to Packt's library of 7,000+ practical books and videos
Feature tick icon Constantly refreshed with 50+ new titles a month
Feature tick icon Exclusive Early access to books as they're written
Feature tick icon Solve problems while you work with advanced search and reference features
Feature tick icon Offline reading on the mobile app
Feature tick icon Choose a DRM-free eBook or Video every month to keep
Feature tick icon PLUS own as many other DRM-free eBooks or Videos as you like for just ₱260 each
Feature tick icon Exclusive print discounts
$279.99 billed in 18 months
Feature tick icon Unlimited access to Packt's library of 7,000+ practical books and videos
Feature tick icon Constantly refreshed with 50+ new titles a month
Feature tick icon Exclusive Early access to books as they're written
Feature tick icon Solve problems while you work with advanced search and reference features
Feature tick icon Offline reading on the mobile app
Feature tick icon Choose a DRM-free eBook or Video every month to keep
Feature tick icon PLUS own as many other DRM-free eBooks or Videos as you like for just ₱260 each
Feature tick icon Exclusive print discounts

Frequently bought together


Stars icon
Total 4,490.98
Learning ELK Stack
₱2500.99
Kibana Essentials
₱1989.99
Total 4,490.98 Stars icon

Table of Contents

11 Chapters
1. Introduction to ELK Stack Chevron down icon Chevron up icon
2. Building Your First Data Pipeline with ELK Chevron down icon Chevron up icon
3. Collect, Parse and Transform Data with Logstash Chevron down icon Chevron up icon
4. Creating Custom Logstash Plugins Chevron down icon Chevron up icon
5. Why Do We Need Elasticsearch in ELK? Chevron down icon Chevron up icon
6. Finding Insights with Kibana Chevron down icon Chevron up icon
7. Kibana – Visualization and Dashboard Chevron down icon Chevron up icon
8. Putting It All Together Chevron down icon Chevron up icon
9. ELK Stack in Production Chevron down icon Chevron up icon
10. Expanding Horizons with ELK Chevron down icon Chevron up icon
Index Chevron down icon Chevron up icon

Customer reviews

Rating distribution
Full star icon Full star icon Full star icon Half star icon Empty star icon 3.2
(5 Ratings)
5 star 20%
4 star 20%
3 star 40%
2 star 0%
1 star 20%
Akshay Aug 24, 2017
Full star icon Full star icon Full star icon Full star icon Full star icon 5
Great to begin ELK
Amazon Verified review Amazon
Amazon Customer Jan 25, 2016
Full star icon Full star icon Full star icon Full star icon Empty star icon 4
Just over a month ago, I purchased a copy of Learning ELK stack and is a good book for those wanting a quick ramp up of what ELK is, This is a good step-by-step guide for Elastic Search, Logstash and Kibana. this book its well written and gives an honest account, i would highly recommend this as a first book read to anyone wanting to know about ELK.
Amazon Verified review Amazon
Kindle Customer Jun 07, 2017
Full star icon Full star icon Full star icon Empty star icon Empty star icon 3
Great quickstart , got me up and going very quickly. It does however lack depth so if you need a reference this would have to be in addition to the elk documentation.
Amazon Verified review Amazon
Mark Grover Jan 09, 2016
Full star icon Full star icon Full star icon Empty star icon Empty star icon 3
A very good book, let down by inaccurate examples of unix command syntax which quite simply doesnt work. I am new to ELK and followed the example scenario presented in the book but had to google and research some of the command syntax as the printed examples returned errors. Its such a shame as the author presents a very thorough step by step guide to configuring the ELK components. I did manage to get everything working and would still recommend the book to newcomers despite the Unix command issues.
Amazon Verified review Amazon
Jascha Casadio Jul 02, 2016
Full star icon Empty star icon Empty star icon Empty star icon Empty star icon 1
The Elasticsearch, Logstash and Kibana trinity, usually referred to as the ELK stack, is by far, the de facto standard in log centralization and analysis. Despite being such a popular solution, with some half a million downloads per month, the titles available to the stack or specific to its components are still very limited. This is partially compensated by the official documentation, which is both friendly and easy to follow and allows anyone to quickly get started. Learning ELK Stack is the only title available, until now, that covers the three products at once. It targets beginners who are interested in an overall view of the stack and its components.Released at the end of 2015, Learning ELK Stack is a short book spanning around 200 pages. As any typical beginner's title, it does start with an introductory chapter that gets the reader through the installation process. What follows is a series of chapters where the author first shows the power of the stack and then dives deeper into its components.The very first chapter already reveals problems that are present throughout the whole book. It is very superficial and does not really get a beginner started. First of all, the stack does require Oracle's JDK installed and configured. This is completely overlooked. Now while someone might argue that it is pretty straightforward through a package manager, the reader can still be using a distro whose package manager installs a version of the JDK that is not suitable for the ELK stack. Likewise, the whole stack can be installed through simple apt commands. The author does cover installation through .tar.gz archives, completley overlooking installing and configuring Java from the source and the default Java (yea, you can have multiple at the same time). Which is not that straightforward.Installation apart, nothing is said about the configuration of the three softwares neither as a standalone nor as a stack. Well, this is not properly correct. The astonishing amount of twelve lines is dedicated to this in fact. After an overall overview of the stack, with an example built using data from Yahoo—you ain't limited to use the stack to process logs, the authors focuses on the components, each at a time. These chapters feel like a reference. Each option is listed, but the examples do not go beyond the two lines. Interestingly, by googling sentences from the book, we find a 70% match analysis with the official documentation (Harvard refers to this as mosaic plagiarism), suggesting a copy/paste with a couple of words added/removed.As an example, this is what we find in Learning ELK stack:The mutate filter is an important filter plugin that helps rename, remove, replace, and modify fields in an incoming event.And this is the information we freely find in the official documentation provided by Elastic:The mutate filter allows you to perform general mutations on fields. You can rename, remove, replace, and modify fields in your events.Tying it all up, I do not really recommend this book ,despite being the only title covering the stack as a whole. The documentation the reader will find is first of all inaccurate. Anything else can be found for free in the official documentation.I have tried to contact Packt Publishing asking what is their position on that matter. I did not get any real answer apart from a semi-automatic one.As usual, you can find more reviews on my personal blog: books.lostinmalloc.com. Feel free to pass by and share your thoughts!
Amazon Verified review Amazon
Get free access to Packt library with over 7500+ books and video courses for 7 days!
Start Free Trial

FAQs

How do I buy and download an eBook? Chevron down icon Chevron up icon

Where there is an eBook version of a title available, you can buy it from the book details for that title. Add either the standalone eBook or the eBook and print book bundle to your shopping cart. Your eBook will show in your cart as a product on its own. After completing checkout and payment in the normal way, you will receive your receipt on the screen containing a link to a personalised PDF download file. This link will remain active for 30 days. You can download backup copies of the file by logging in to your account at any time.

If you already have Adobe reader installed, then clicking on the link will download and open the PDF file directly. If you don't, then save the PDF file on your machine and download the Reader to view it.

Please Note: Packt eBooks are non-returnable and non-refundable.

Packt eBook and Licensing When you buy an eBook from Packt Publishing, completing your purchase means you accept the terms of our licence agreement. Please read the full text of the agreement. In it we have tried to balance the need for the ebook to be usable for you the reader with our needs to protect the rights of us as Publishers and of our authors. In summary, the agreement says:

  • You may make copies of your eBook for your own use onto any machine
  • You may not pass copies of the eBook on to anyone else
How can I make a purchase on your website? Chevron down icon Chevron up icon

If you want to purchase a video course, eBook or Bundle (Print+eBook) please follow below steps:

  1. Register on our website using your email address and the password.
  2. Search for the title by name or ISBN using the search option.
  3. Select the title you want to purchase.
  4. Choose the format you wish to purchase the title in; if you order the Print Book, you get a free eBook copy of the same title. 
  5. Proceed with the checkout process (payment to be made using Credit Card, Debit Cart, or PayPal)
Where can I access support around an eBook? Chevron down icon Chevron up icon
  • If you experience a problem with using or installing Adobe Reader, the contact Adobe directly.
  • To view the errata for the book, see www.packtpub.com/support and view the pages for the title you have.
  • To view your account details or to download a new copy of the book go to www.packtpub.com/account
  • To contact us directly if a problem is not resolved, use www.packtpub.com/contact-us
What eBook formats do Packt support? Chevron down icon Chevron up icon

Our eBooks are currently available in a variety of formats such as PDF and ePubs. In the future, this may well change with trends and development in technology, but please note that our PDFs are not Adobe eBook Reader format, which has greater restrictions on security.

You will need to use Adobe Reader v9 or later in order to read Packt's PDF eBooks.

What are the benefits of eBooks? Chevron down icon Chevron up icon
  • You can get the information you need immediately
  • You can easily take them with you on a laptop
  • You can download them an unlimited number of times
  • You can print them out
  • They are copy-paste enabled
  • They are searchable
  • There is no password protection
  • They are lower price than print
  • They save resources and space
What is an eBook? Chevron down icon Chevron up icon

Packt eBooks are a complete electronic version of the print edition, available in PDF and ePub formats. Every piece of content down to the page numbering is the same. Because we save the costs of printing and shipping the book to you, we are able to offer eBooks at a lower cost than print editions.

When you have purchased an eBook, simply login to your account and click on the link in Your Download Area. We recommend you saving the file to your hard drive before opening it.

For optimal viewing of our eBooks, we recommend you download and install the free Adobe Reader version 9.