Search icon CANCEL
Subscription
0
Cart icon
Your Cart (0 item)
Close icon
You have no products in your basket yet
Save more on your purchases! discount-offer-chevron-icon
Savings automatically calculated. No voucher code required.
Arrow left icon
Explore Products
Best Sellers
New Releases
Books
Videos
Audiobooks
Learning Hub
Newsletter Hub
Free Learning
Arrow right icon
timer SALE ENDS IN
0 Days
:
00 Hours
:
00 Minutes
:
00 Seconds
Learning ELK Stack
Learning ELK Stack

Learning ELK Stack: Build mesmerizing visualizations, analytics, and logs from your data using Elasticsearch, Logstash, and Kibana

eBook
$35.98 $39.99
Paperback
$48.99
Subscription
Free Trial
Renews at $19.99p/m

What do you get with a Packt Subscription?

Free for first 7 days. $19.99 p/m after that. Cancel any time!
Product feature icon Unlimited ad-free access to the largest independent learning library in tech. Access this title and thousands more!
Product feature icon 50+ new titles added per month, including many first-to-market concepts and exclusive early access to books as they are being written.
Product feature icon Innovative learning tools, including AI book assistants, code context explainers, and text-to-speech.
Product feature icon Thousands of reference materials covering every tech concept you need to stay up to date.
Subscribe now
View plans & pricing
Table of content icon View table of contents Preview book icon Preview Book

Learning ELK Stack

Chapter 2. Building Your First Data Pipeline with ELK

In the previous chapter, we got familiar with each component of ELK Stack—Elasticsearch, Logstash, and Kibana. We got the components installed and configured. In this chapter, we will build our first basic data pipeline using ELK Stack. This will help us understand how easy it is to get together the components of ELK Stack to build an end-to-end analytics pipeline.

While running the example in this chapter, we assume that you already installed Elasticsearch, Logstash, and Kibana as described in Chapter 1, Introduction to ELK Stack.

Input dataset

For our example, the dataset that we are going to use here is the daily Google (GOOG) Quotes price dataset over a 6 month period from July 1, 2014 to December 31, 2014. This is a good dataset to understand how we can quickly analyze simple datasets, such as these, with ELK.

Note

This dataset can be easily downloaded from the following source:

http://finance.yahoo.com/q/hp?s=GOOG

Data format for input dataset

The most significant fields of this dataset are Date, Open Price, Close Price, High Price, Volume, and Adjusted Price.

The following table shows some of the sample data from the dataset. The actual dataset is in the CSV format.

Date

Open

High

Low

Close

Volume

Adj Close

Dec 31, 2014

531.25

532.60

525.80

526.40

1,368,200

526.40

Dec 30, 2014

528.09

531.15

527.13

530.42

876,300

530.42

Dec 29, 2014

532.19

535.48

530.01

530.33

2,278,500

530.33

Dec 26, 2014

528.77

534.25

527.31

534.03

1,036,000

534.03

Dec 24, 2014

530.51

531.76

527.02

528.77...

Configuring Logstash input

As we already know, Logstash has a rich set of plugins for different types of inputs, outputs and filters, which can read, parse, and filter data as per our needs. We will utilize the file input plugin to read the source file.

A file input plugin streams events from the input file, and each event is assumed as a single line. It automatically detects file rotation and handles it. It maintains the location where it left reading, and will automatically detect the new data if configured correctly. It reads files in a similar manner:

tail -0f 

In general, a file input plugin configuration will look as follows:

input {
 
file {
    path => #String (path of the files) (required) 
    start_position => #String (optional, default "end")
    tags => #array (optional)
    type => #string (optional)
}

}
  • path: The path field is the only required field in file input plugin, which represents the path of the file from where input events have to be processed...

Filtering and processing input

Once we configure the input file, we need to filter the input based on our needs so that we can identify which fields we need, and process them as per the required analysis.

A filter plugin will perform the intermediary processing on the input event. We can apply the filter conditionally based on certain fields.

Since our input file is a CSV file, we will use the csv filter for the same. The csv filter takes an event field that contains CSV formatted data, parses it, and stores it as individual fields. It can also parse data with any separator other than commas. A typical csv filter is as follows:

filter {  
    csv {
        columns => #Array of column names.
        separator => #String ; default -","
    }
}

The attribute columns take the name of fields in our CSV file, which is optional. By default, the columns will be named as column 1, column 2, and so on.

The attribute separator defines what character is used to separate the different columns...

Putting data to Elasticsearch

Now that we have set up the data to be consumed by a CSV file into Logstash, followed by parsing and processing based on the data type needed, we now need to put the data in Elasticsearch so that we can index the different fields and consume them later via the Kibana interface.

We will use the output plugin of Logstash for an elasticsearch output.

A typical elasticsearch plugin configuration looks like this:

output {

  elasticsearch {

    action =>  # string (optional), default: "index"

    cluster =>  # string (optional)

    host =>  # string (optional)
   
    document_id =>  # string (optional), default: nil


    index =>  # string (optional), default: "logstash-%{+YYYY.MM.dd}"
    index_type =>  # string (optional)
    port =>  # string (optional)
    protocol =>  # string, one of ["node", "transport", "http"] (optional)
  }
}
  • action: This specifies what action to perform on incoming...

Visualizing with Kibana

Now when you verify that your data is indexed successfully in Elasticsearch, we can go ahead and look at the Kibana interface to get some useful analytics from the data.

Running Kibana

As described in the previous chapter, we will start the Kibana service from the Kibana installation directory.

$ bin/kibana

Now, let's see Kibana up and running similar to the following screenshot on the browser, by going to the following URL:

http://localhost:5601

Running Kibana

Kibana Discover page

As we already set up Kibana to take logstash-* indexes by default, it displays the indexed data as a histogram of counts, and the associated data as fields in the JSON format.

First of all, we need to set the date filter to filter based on our date range so that we can build our analysis on the same. Since we took data from July 1, 2014 to December 31, 2014, we will configure our date filter for the same.

Clicking on the Time Filter icon at the extreme top-right corner, we can set an Absolute Time Filter...

Input dataset


For our example, the dataset that we are going to use here is the daily Google (GOOG) Quotes price dataset over a 6 month period from July 1, 2014 to December 31, 2014. This is a good dataset to understand how we can quickly analyze simple datasets, such as these, with ELK.

Note

This dataset can be easily downloaded from the following source:

http://finance.yahoo.com/q/hp?s=GOOG

Data format for input dataset

The most significant fields of this dataset are Date, Open Price, Close Price, High Price, Volume, and Adjusted Price.

The following table shows some of the sample data from the dataset. The actual dataset is in the CSV format.

Date

Open

High

Low

Close

Volume

Adj Close

Dec 31, 2014

531.25

532.60

525.80

526.40

1,368,200

526.40

Dec 30, 2014

528.09

531.15

527.13

530.42

876,300

530.42

Dec 29, 2014

532.19

535.48

530.01

530.33

2,278,500

530.33

Dec 26, 2014

528.77

534.25

527.31

534.03

1,036,000

534.03

Dec 24, 2014

530.51

531.76

527.02

528.77

...
Left arrow icon Right arrow icon

Description

The ELK stack—Elasticsearch, Logstash, and Kibana, is a powerful combination of open source tools. Elasticsearch is for deep search and data analytics. Logstash is for centralized logging, log enrichment, and parsing. Kibana is for powerful and beautiful data visualizations. In short, the Elasticsearch ELK stack makes searching and analyzing data easier than ever before. This book will introduce you to the ELK (Elasticsearch, Logstash, and Kibana) stack, starting by showing you how to set up the stack by installing the tools, and basic configuration. You’ll move on to building a basic data pipeline using the ELK stack. Next, you’ll explore the key features of Logstash and its role in the ELK stack, including creating Logstash plugins, which will enable you to use your own customized plugins. The importance of Elasticsearch and Kibana in the ELK stack is also covered, along with various types of advanced data analysis, and a variety of charts, tables ,and maps. Finally, by the end of the book you will be able to develop full-fledged data pipeline using the ELK stack and have a solid understanding of the role of each of the components.

Who is this book for?

If you are a developer or DevOps engineer interested in building a system that provides amazing insights and business metrics out of data sources, of various formats and types, using the open source technology stack that ELK provides, then this book is for you. Basic knowledge of Unix or any programming language will be helpful to make the most out of this book.

What you will learn

  • Install, configure, and run Elasticsearch, Logstash, and Kibana
  • Understand the need for log analytics and the current challenges in log analysis
  • Build your own data pipeline using the ELK stack
  • Familiarize yourself with the key features of Logstash and the variety of input, filter, and output plugins it provides
  • Build your own custom Logstash plugin
  • Create actionable insights using charts, histograms, and quick search features in Kibana4
  • Understand the role of Elasticsearch in the ELK stack

Product Details

Country selected
Publication date, Length, Edition, Language, ISBN-13
Publication date : Nov 26, 2015
Length: 206 pages
Edition : 1st
Language : English
ISBN-13 : 9781785887154
Vendor :
Elastic
Category :

What do you get with a Packt Subscription?

Free for first 7 days. $19.99 p/m after that. Cancel any time!
Product feature icon Unlimited ad-free access to the largest independent learning library in tech. Access this title and thousands more!
Product feature icon 50+ new titles added per month, including many first-to-market concepts and exclusive early access to books as they are being written.
Product feature icon Innovative learning tools, including AI book assistants, code context explainers, and text-to-speech.
Product feature icon Thousands of reference materials covering every tech concept you need to stay up to date.
Subscribe now
View plans & pricing

Product Details

Publication date : Nov 26, 2015
Length: 206 pages
Edition : 1st
Language : English
ISBN-13 : 9781785887154
Vendor :
Elastic
Category :

Packt Subscriptions

See our plans and pricing
Modal Close icon
$19.99 billed monthly
Feature tick icon Unlimited access to Packt's library of 7,000+ practical books and videos
Feature tick icon Constantly refreshed with 50+ new titles a month
Feature tick icon Exclusive Early access to books as they're written
Feature tick icon Solve problems while you work with advanced search and reference features
Feature tick icon Offline reading on the mobile app
Feature tick icon Simple pricing, no contract
$199.99 billed annually
Feature tick icon Unlimited access to Packt's library of 7,000+ practical books and videos
Feature tick icon Constantly refreshed with 50+ new titles a month
Feature tick icon Exclusive Early access to books as they're written
Feature tick icon Solve problems while you work with advanced search and reference features
Feature tick icon Offline reading on the mobile app
Feature tick icon Choose a DRM-free eBook or Video every month to keep
Feature tick icon PLUS own as many other DRM-free eBooks or Videos as you like for just $5 each
Feature tick icon Exclusive print discounts
$279.99 billed in 18 months
Feature tick icon Unlimited access to Packt's library of 7,000+ practical books and videos
Feature tick icon Constantly refreshed with 50+ new titles a month
Feature tick icon Exclusive Early access to books as they're written
Feature tick icon Solve problems while you work with advanced search and reference features
Feature tick icon Offline reading on the mobile app
Feature tick icon Choose a DRM-free eBook or Video every month to keep
Feature tick icon PLUS own as many other DRM-free eBooks or Videos as you like for just $5 each
Feature tick icon Exclusive print discounts

Frequently bought together


Stars icon
Total $ 87.98
Learning ELK Stack
$48.99
Kibana Essentials
$38.99
Total $ 87.98 Stars icon

Table of Contents

11 Chapters
1. Introduction to ELK Stack Chevron down icon Chevron up icon
2. Building Your First Data Pipeline with ELK Chevron down icon Chevron up icon
3. Collect, Parse and Transform Data with Logstash Chevron down icon Chevron up icon
4. Creating Custom Logstash Plugins Chevron down icon Chevron up icon
5. Why Do We Need Elasticsearch in ELK? Chevron down icon Chevron up icon
6. Finding Insights with Kibana Chevron down icon Chevron up icon
7. Kibana – Visualization and Dashboard Chevron down icon Chevron up icon
8. Putting It All Together Chevron down icon Chevron up icon
9. ELK Stack in Production Chevron down icon Chevron up icon
10. Expanding Horizons with ELK Chevron down icon Chevron up icon
Index Chevron down icon Chevron up icon

Customer reviews

Rating distribution
Full star icon Full star icon Full star icon Half star icon Empty star icon 3.2
(5 Ratings)
5 star 20%
4 star 20%
3 star 40%
2 star 0%
1 star 20%
Akshay Aug 24, 2017
Full star icon Full star icon Full star icon Full star icon Full star icon 5
Great to begin ELK
Amazon Verified review Amazon
Amazon Customer Jan 25, 2016
Full star icon Full star icon Full star icon Full star icon Empty star icon 4
Just over a month ago, I purchased a copy of Learning ELK stack and is a good book for those wanting a quick ramp up of what ELK is, This is a good step-by-step guide for Elastic Search, Logstash and Kibana. this book its well written and gives an honest account, i would highly recommend this as a first book read to anyone wanting to know about ELK.
Amazon Verified review Amazon
Kindle Customer Jun 07, 2017
Full star icon Full star icon Full star icon Empty star icon Empty star icon 3
Great quickstart , got me up and going very quickly. It does however lack depth so if you need a reference this would have to be in addition to the elk documentation.
Amazon Verified review Amazon
Mark Grover Jan 09, 2016
Full star icon Full star icon Full star icon Empty star icon Empty star icon 3
A very good book, let down by inaccurate examples of unix command syntax which quite simply doesnt work. I am new to ELK and followed the example scenario presented in the book but had to google and research some of the command syntax as the printed examples returned errors. Its such a shame as the author presents a very thorough step by step guide to configuring the ELK components. I did manage to get everything working and would still recommend the book to newcomers despite the Unix command issues.
Amazon Verified review Amazon
Jascha Casadio Jul 02, 2016
Full star icon Empty star icon Empty star icon Empty star icon Empty star icon 1
The Elasticsearch, Logstash and Kibana trinity, usually referred to as the ELK stack, is by far, the de facto standard in log centralization and analysis. Despite being such a popular solution, with some half a million downloads per month, the titles available to the stack or specific to its components are still very limited. This is partially compensated by the official documentation, which is both friendly and easy to follow and allows anyone to quickly get started. Learning ELK Stack is the only title available, until now, that covers the three products at once. It targets beginners who are interested in an overall view of the stack and its components.Released at the end of 2015, Learning ELK Stack is a short book spanning around 200 pages. As any typical beginner's title, it does start with an introductory chapter that gets the reader through the installation process. What follows is a series of chapters where the author first shows the power of the stack and then dives deeper into its components.The very first chapter already reveals problems that are present throughout the whole book. It is very superficial and does not really get a beginner started. First of all, the stack does require Oracle's JDK installed and configured. This is completely overlooked. Now while someone might argue that it is pretty straightforward through a package manager, the reader can still be using a distro whose package manager installs a version of the JDK that is not suitable for the ELK stack. Likewise, the whole stack can be installed through simple apt commands. The author does cover installation through .tar.gz archives, completley overlooking installing and configuring Java from the source and the default Java (yea, you can have multiple at the same time). Which is not that straightforward.Installation apart, nothing is said about the configuration of the three softwares neither as a standalone nor as a stack. Well, this is not properly correct. The astonishing amount of twelve lines is dedicated to this in fact. After an overall overview of the stack, with an example built using data from Yahoo—you ain't limited to use the stack to process logs, the authors focuses on the components, each at a time. These chapters feel like a reference. Each option is listed, but the examples do not go beyond the two lines. Interestingly, by googling sentences from the book, we find a 70% match analysis with the official documentation (Harvard refers to this as mosaic plagiarism), suggesting a copy/paste with a couple of words added/removed.As an example, this is what we find in Learning ELK stack:The mutate filter is an important filter plugin that helps rename, remove, replace, and modify fields in an incoming event.And this is the information we freely find in the official documentation provided by Elastic:The mutate filter allows you to perform general mutations on fields. You can rename, remove, replace, and modify fields in your events.Tying it all up, I do not really recommend this book ,despite being the only title covering the stack as a whole. The documentation the reader will find is first of all inaccurate. Anything else can be found for free in the official documentation.I have tried to contact Packt Publishing asking what is their position on that matter. I did not get any real answer apart from a semi-automatic one.As usual, you can find more reviews on my personal blog: books.lostinmalloc.com. Feel free to pass by and share your thoughts!
Amazon Verified review Amazon
Get free access to Packt library with over 7500+ books and video courses for 7 days!
Start Free Trial

FAQs

What is included in a Packt subscription? Chevron down icon Chevron up icon

A subscription provides you with full access to view all Packt and licnesed content online, this includes exclusive access to Early Access titles. Depending on the tier chosen you can also earn credits and discounts to use for owning content

How can I cancel my subscription? Chevron down icon Chevron up icon

To cancel your subscription with us simply go to the account page - found in the top right of the page or at https://subscription.packtpub.com/my-account/subscription - From here you will see the ‘cancel subscription’ button in the grey box with your subscription information in.

What are credits? Chevron down icon Chevron up icon

Credits can be earned from reading 40 section of any title within the payment cycle - a month starting from the day of subscription payment. You also earn a Credit every month if you subscribe to our annual or 18 month plans. Credits can be used to buy books DRM free, the same way that you would pay for a book. Your credits can be found in the subscription homepage - subscription.packtpub.com - clicking on ‘the my’ library dropdown and selecting ‘credits’.

What happens if an Early Access Course is cancelled? Chevron down icon Chevron up icon

Projects are rarely cancelled, but sometimes it's unavoidable. If an Early Access course is cancelled or excessively delayed, you can exchange your purchase for another course. For further details, please contact us here.

Where can I send feedback about an Early Access title? Chevron down icon Chevron up icon

If you have any feedback about the product you're reading, or Early Access in general, then please fill out a contact form here and we'll make sure the feedback gets to the right team. 

Can I download the code files for Early Access titles? Chevron down icon Chevron up icon

We try to ensure that all books in Early Access have code available to use, download, and fork on GitHub. This helps us be more agile in the development of the book, and helps keep the often changing code base of new versions and new technologies as up to date as possible. Unfortunately, however, there will be rare cases when it is not possible for us to have downloadable code samples available until publication.

When we publish the book, the code files will also be available to download from the Packt website.

How accurate is the publication date? Chevron down icon Chevron up icon

The publication date is as accurate as we can be at any point in the project. Unfortunately, delays can happen. Often those delays are out of our control, such as changes to the technology code base or delays in the tech release. We do our best to give you an accurate estimate of the publication date at any given time, and as more chapters are delivered, the more accurate the delivery date will become.

How will I know when new chapters are ready? Chevron down icon Chevron up icon

We'll let you know every time there has been an update to a course that you've bought in Early Access. You'll get an email to let you know there has been a new chapter, or a change to a previous chapter. The new chapters are automatically added to your account, so you can also check back there any time you're ready and download or read them online.

I am a Packt subscriber, do I get Early Access? Chevron down icon Chevron up icon

Yes, all Early Access content is fully available through your subscription. You will need to have a paid for or active trial subscription in order to access all titles.

How is Early Access delivered? Chevron down icon Chevron up icon

Early Access is currently only available as a PDF or through our online reader. As we make changes or add new chapters, the files in your Packt account will be updated so you can download them again or view them online immediately.

How do I buy Early Access content? Chevron down icon Chevron up icon

Early Access is a way of us getting our content to you quicker, but the method of buying the Early Access course is still the same. Just find the course you want to buy, go through the check-out steps, and you’ll get a confirmation email from us with information and a link to the relevant Early Access courses.

What is Early Access? Chevron down icon Chevron up icon

Keeping up to date with the latest technology is difficult; new versions, new frameworks, new techniques. This feature gives you a head-start to our content, as it's being created. With Early Access you'll receive each chapter as it's written, and get regular updates throughout the product's development, as well as the final course as soon as it's ready.We created Early Access as a means of giving you the information you need, as soon as it's available. As we go through the process of developing a course, 99% of it can be ready but we can't publish until that last 1% falls in to place. Early Access helps to unlock the potential of our content early, to help you start your learning when you need it most. You not only get access to every chapter as it's delivered, edited, and updated, but you'll also get the finalized, DRM-free product to download in any format you want when it's published. As a member of Packt, you'll also be eligible for our exclusive offers, including a free course every day, and discounts on new and popular titles.