What do you get with Print?

Instant access to your digital copy whilst your Print order is Shipped

Paperback book shipped to your preferred address

Redeem a companion digital copy on all Print orders

Access this title in our online reader with advanced features

DRM FREE - Read whenever, wherever and however you want

Splunk Operational Intelligence Cookbook

Chapter 1. Play Time – Getting Data In

In this chapter, we will cover the basic ways to get data into Splunk. You will learn about:

Indexing files and directories
Getting data through network ports
Using scripted inputs
Using modular inputs
Using the Universal Forwarder to gather data
Loading the sample data for this book
Defining field extractions
Defining event types and tags

Introduction

The machine data that facilitates operational intelligence comes in many different forms and from many different sources. Splunk is able to collect and index data from many different sources, including logfiles written by web servers or business applications, syslog data streaming in from network devices, or the output of custom developed scripts. Even data that looks complex at first can be easily collected, indexed, transformed, and presented back to you in real time.

This chapter will walk you through the basic recipes that will act as the building blocks to get the data you want into Splunk. The chapter will further serve as an introduction to the sample datasets that we will use to build our own Operational Intelligence Splunk app. The datasets will be coming from a hypothetical, three-tier, e-commerce web application and will contain web server logs, application logs, and database logs.

Splunk Enterprise can index any type of data; however, it works best with time-series data (data with timestamps). When Splunk Enterprise indexes data, it breaks it into events, based on timestamps and/or event size, and puts them into indexes. Indexes are data stores that Splunk has engineered to be very fast, searchable, and scalable across a distributed server environment; they are commonly referred to as indexers. This is also why we refer to the data being put into Splunk as being indexed.

All data indexed into Splunk is assigned a source type. The source type helps identify the data format type of the event and where it has come from. Splunk has a number of preconfigured source types, but you can also specify your own. The example sourcetypes include access_combined, cisco_syslog, and linux_secure. The source type is added to the data when the indexer indexes it into Splunk. It is a key field that is used when performing field extractions and in many searches to filter the data being searched.

The Splunk community plays a big part in making it easy to get data into Splunk. The ability to extend Splunk has provided the opportunity for the development of inputs, commands, and applications that can be easily shared. If there is a particular system or application you are looking to index data from, there is most likely someone who has developed and published relevant configurations and tools that can be easily leveraged by your own Splunk Enterprise deployment.

Splunk Enterprise is designed to make the collection of data very easy, and it will not take long before you are being asked or you yourself try to get as much data into Splunk as possible—at least as much as your license will allow for!

Indexing files and directories

File- and directory-based inputs are the most commonly used ways of getting data into Splunk. The primary need for these types of inputs will be to index logfiles. Almost every application or system will produce a logfile, and it is generally full of data that you would want to be able to search and report on.

Splunk is able to continuously monitor for new data being written to existing files or new files added to a directory, and it is able to index this data in real time. Depending on the type of application that creates the logfiles, you would set up Splunk to either monitor an individual file based on its location or scan an entire directory and monitor all the files that exist within it. The later configuration is more commonly used when the logfiles being produced have unique filenames, for example, the name they have contains a timestamp.

This recipe will show you how to configure Splunk to continuously monitor and index the contents of a rolling logfile located on the Splunk server. The recipe specifically shows how to monitor and index the Linux system's messages logfile (/var/log/messages). However, the same principle can be applied to a logfile on a Windows system, and a sample file is provided. Do not attempt to index the Windows event logs this way, as Splunk has specific Windows event inputs for this.

Getting ready

To step through this recipe, you will need a running Splunk Enterprise server and access to read the /var/log/messages file on Linux. There are no other prerequisites. If you are not using Linux and/or do not have access to the /var/log/messages location on your Splunk server, please use the cp01_messages.log file that is provided and upload it to an accessible directory on your Splunk server.

Tip

Downloading the example code

You can download the example code files for all Packt books you have purchased from your account at http://www.packtpub.com. If you purchased this book elsewhere, you can visit http://www.packtpub.com/support and register to have the files e-mailed directly to you.

How to do it...

Follow the steps in the recipe to monitor and index the contents of a file:

Log in to your Splunk server.
From the home launcher in the top-right corner, click on the Add Data button.
In the Choose a Data Type list, click on A file or directory of files.
Click on Next in the Consume any file on this Splunk server option.
Select Preview data before indexing and enter the path to the logfile (/var/log/messages or the location of the cp01_messages.log file) and click on Continue.
Select Start a new source type and click on Continue.
Assuming that you are using the provided file or the native /var/log/messages file, the data preview will show the correct line breaking of events and timestamp recognition. Click on the Continue button.
A Review settings box will pop up. Enter linux_messages as the source type and then, click on Save source type.
A Sourcetype saved box will appear. Select Create input.
In the Source section, select Continuously index data from a file or directory this Splunk instance can access and fill in the path to your data.
Tip
If you are just looking to do a one-time upload of a file, you can select Upload and Index a file instead. This can be useful to index a set of data that you would like to put into Splunk, either to backfill some missing or incomplete data or just to take advantage of its searching and reporting tools.
Ignore the other settings for now and simply click on Save. Then, on the next screen, click on Start searching. In the search bar, enter the following search over a time range of All time:
```
 sourcetype=linux_messages
```
Tip
In this recipe, we could have simply used the common syslog source type; however, starting a new source type is often a better choice. The syslog format can look completely different depending on the data source. As knowledge objects, such as field extractions, are built on top of source types, using a single syslog source type for everything can make it challenging to search for the data you need.

How it works...

When you add a new file or directory data input, you are basically adding a new configuration stanza into an inputs.conf file behind the scenes. The Splunk server can contain one or more inputs.conf files, and these files are either located in $SPLUNK_HOME/etc/system/local or in the local directory of a Splunk app.

Splunk uses the monitor input type and is set to point to either a file or a directory. If you set the monitor to a directory, all files within that directory will be monitored. When Splunk monitors files, it initially starts by indexing all of the data that it can read from the beginning. Once complete, Splunk will maintain a record of where it last read data from, and if any new data comes into the file, it will read this data and advance the record. The process is nearly identical to using the tail command in Unix-based operating systems. If you are monitoring a directory, Splunk also provides many additional configuration options such as blacklisting files you don't want Splunk to index.

Note

For more information on Splunk's configuration files, visit http://docs.splunk.com/Documentation/Splunk/latest/Admin/Aboutconfigurationfiles.

There's more...

While adding inputs to monitor files and directories can be done through the web interface of Splunk as outlined in this recipe, there are other approaches to add multiple inputs quickly. These allow for customization of the many configuration options that Splunk provides.

Adding a file or directory data input via the CLI

Instead of going via the GUI, you could add a file or directory input via the Splunk CLI (command-line interface). Navigate to your $SPLUNK_HOME/bin directory and execute the following command (replacing the file or directory to be monitored with your own):

For Unix:

./splunk add monitor /var/log/messages –sourcetype linux_messages

For Windows:

splunk add monitor c:\filelocation\cp01_messages.log –sourcetype linux_messages

There are a number of different parameters that can be passed along with the file location to monitor. See the Splunk documentation for more on data inputs using the CLI (http://docs.splunk.com/Documentation/Splunk/latest/Data/MonitorfilesanddirectoriesusingtheCLI).

Adding a file or directory input via inputs.conf

Another common method of adding file and directory inputs is to manually add them to the inputs.conf configuration file directly. This approach is often used for large environments or when configuring Splunk forwarders to monitor for files or directories on endpoints.

Edit $SPLUNK_HOME/etc/system/local/inputs.conf and add your input. After your inputs are added, Splunk will need to be restarted to recognize these changes:

For Unix:

[monitor:///var/log/messages]
sourcetype = linux_messages

For Windows:

[monitor://c:\filelocation\cp01_messages.log]
sourcetype = linux_messages

Tip

Editing inputs.conf directly is often a much faster way of adding new files or directories to monitor when several inputs are needed. When editing inputs.conf, ensure that the correct syntax is used and remember that Splunk will need a restart for modifications to take effect. Additionally, specifying the source type in the inputs.conf file is the best practice to assign source types.

One-time indexing of data files via the Splunk CLI

Although you can select Upload and Index a file from the Splunk GUI to upload and index a file, there are a couple of CLI functions that can be used to perform one-time bulk loads of data.

Use the oneshot command to tell Splunk where the file is located and which parameters to use, such as the source type:

./splunk add oneshot XXXXXXX

Another way is to place the file you wish to index into the Splunk spool directory, $SPLUNK_HOME/var/spool/splunk, and then add the file using the spool command:

./splunk spool XXXXXXX

Tip

If using Windows, omit ./ that is in front of the Splunk commands, mentioned earlier.

Indexing the Windows event logs

Splunk comes with special inputs.conf configurations for some source types, including monitoring the Windows event logs. Typically, the Splunk Universal Forwarder (UF) would be installed on a Windows server and configured to forward the Windows events to the Splunk indexer(s). The configurations for inputs.conf to monitor Windows security, application, and system event logs in real time are as follows:

[WinEventLog://Application]
disabled = 0 
[WinEventLog://Security]
disabled = 0 
[WinEventLog://System]
disabled = 0

By default, the event data will go into the main index, unless another index is specified.

I have had a keen interest in Splunk for over 2 years and have seen the product grown and evolve from 4.2 to the current release of 6.2. Coming from the IT/Legal Compliance background, I have found the product at times challenging (learning regular expressions... anyone?). When Splunk Operational Intelligence came out for pre-order, it quickly caught my eye and was quick to add it to my wish list.This latest publication has been a real blessing. It includes the latest features within Splunk, like pivot tables and includes steps on how to build your own applications. So far I have found the book very helpful. I like how the information is arranged and how the how-to steps are presented in a very visual format. The author's use of examples are also very helpful and very easy to implement in any environment. To be fair I am only halfway through this book. But will post an update when I have completed it.My recommendation is that if you are new to Splunk, then get this book. It will help you get up to speed on the product and can help you navigate the extensive material provided by Splunk. If you are an experienced Splunker, some of the book's contents maybe redundant, but even so a walk through on some of the new features cant hurt. I've found a few useful nuggets of information as I worked through the book. I hope this review helps. Go get the latest version from splunk.com, get this book and get splunking!Disclaimer, I was fortunate enough to receive a electronic copy of this book by the publisher. My review was independent of this as I was about to order to the ebook this week. That said, I'll probably will buy the hard back copy because I have found it to be really helpful. I hope the author continues to provided revised editions as the product continues to evolve and perhaps develop a book more geared for Splunk deployments, the clustering of Indexers and SplunkWeb servers and walking the reader on setting up SSO Proxy servers using IIS and Apache.

BENJAMIN AFOLABI Dec 01, 2014

One of the best Splunk book available in the market..This will definitely get you started and even help advance users.

Lee Gorby Jul 05, 2016

Really good for beginners!

Splunk Operational Intelligence Cookbook: With Splunk, reporting and communicating insight is simple – find out with this Splunk book, created to help you unlock more effective Business Intelligence

What do you get with Print?

Splunk Operational Intelligence Cookbook

Chapter 1. Play Time – Getting Data In

Introduction

Indexing files and directories

Getting ready

Tip

How to do it...

Tip

Tip

How it works...

Note

There's more...

Adding a file or directory data input via the CLI

Adding a file or directory input via inputs.conf

Tip

One-time indexing of data files via the Splunk CLI

Tip

Indexing the Windows event logs

See also

Page 1 of 11

Description

Product Details

What do you get with Print?

Product Details

Frequently bought together

Table of Contents

Recommendations for you

Customer reviews

Filter reviews by

People who bought this also bought

FAQs

Splunk Operational Intelligence Cookbook: With Splunk, reporting and communicating insight is simple – find out with this Splunk book, created to help you unlock more effective Business Intelligence

What do you get with Print?

Contact Details

Shipping Address

Billing Address

Tip

Tip

Adding a file or directory data input via the CLI

Adding a file or directory input via inputs.conf

One-time indexing of data files via the Splunk CLI

Indexing the Windows event logs

Description

Product Details

What do you get with Print?

Contact Details

Shipping Address

Billing Address

Product Details

Packt Subscriptions

Frequently bought together

Table of Contents

Recommendations for you

Customer reviews

Filter reviews by

People who bought this also bought

FAQs