Splunk 7.x Quick Start Guide

Introduction to Splunk

Welcome to the first chapter of the Splunk 7.x Quick Start Guide! This chapter introduces Splunk to the newcomer and guides them progressively toward understanding the reasons why Splunk is so popular. It introduces all the powerful capabilities and solutions it offers for collecting and analyzing machine data from a wide variety of devices and environments. This chapter also includes a high-level overview of how Splunk works to serve as a foundation for digging into more details in the chapters to come.

The topics that are covered in this chapter include the following:

Understanding what Splunk is and what problems it solves
Installing a free version of Splunk Enterprise
Becoming familiar with the major components of a Splunk solution and their functions
Becoming aware of the major processing tiers of a Splunk deployment—data input, parsing, indexing, and search
Learning about the four key Splunk fields for every event—_time, host, source, and sourcetype—and why they're important
Becoming aware of the Splunk community and all the information and resources available to learn more about configuring and using Splunk

What is Splunk?

Okay—so what is Splunk, anyway? How do you explain this product to your peers, friends, and family in a way that is easy to comprehend without watering down its awesome capabilities? Here's how I explain it, with an introductory setting and then increasing levels of powerful uses—until I notice their eyes starting to glaze over, at which point I stop and summarize again: "It's like Google for all kinds of machine data!"

Every company will have tens, hundreds, or maybe thousands of application and web servers, databases, and network devices such as switches, routers, and firewalls; all kinds of sensors, and so on—and all of these create log files or data streams that record their activities and statuses over time. Now, imagine needing to troubleshoot a problem that might be caused by any one of several parts of a system, and having to log into each of these machines one at a time, manually dig through its log file looking for clues, then log into the next, and so on—you can see how tedious and time-consuming this can become. Or maybe you want to monitor critical processes to make sure things are running well—how do you do that for a lot of machines?

Splunk is a software platform that collects and stores all this machine data in one place. It makes it as easy to search through and investigate that data as using Google. Basically, it's Google for log files! Beyond troubleshooting, you can use this search capability to build reports and dashboards to monitor performance, reliability, or other metrics across a whole collection of related servers and devices, and even create alerts to warn you by text or email when something is going wrong. It's also used to detect security threats, and since you have all this data in one place, you can do event correlation across devices and apply machine learning to it for the purposes of anomaly detection, user behavior analytics, and even predictive analytics to identify potential problems before they happen.

Splunk has a media kit brochure that covers the spectrum of ways Splunk helps companies extract value from their machine data, which can be found at: https://www.splunk.com/en_us/newsroom/media-kit.html.

The following diagram illustrates the spectrum of data you can collect with Splunk, and captures the essence of what Splunk does:

Splunk data sources and use cases

Splunk products

There are several types and licensing models of Splunk available to suit the needs of its customer base.

Splunk Enterprise is designed for on-premise deployments; it can be scaled to support an unlimited number of users and ingestion volumes by adding the necessary number and types of Splunk functional software components (indexers and search heads) on customer-supplied servers. The cost of a Splunk Enterprise license is based on daily ingestion volume. A wide range of applications for Splunk Enterprise, written by both Splunk and the user community, that add value to the product are available for free from the Splunkbase website. Splunk also sells several sophisticated premium solution applications such as IT Service Intelligence, Enterprise Security, and User Behavior Analytics on an individual license basis, and a Machine Learning Toolkit is available for free from Splunkbase if you want to roll out your own ML solutions.

Splunk Cloud is a cloud-based software as a service (SaaS) version of Splunk Enterprise; it is also licensed based on daily ingestion volume, and offers all the functionality of the on-premise Splunk Enterprise product. The core Splunk infrastructure is provided and managed by Splunk, while data inputs, approved apps, reports, dashboards, alerts, and so on are managed by the customer.

Splunk Light is designed to be a small-scale solution. It allows up to 20 GB/day of ingestion volume, five users, and reports/dashboards/alerts, but does not provide support for distributed deployments, Splunkbase or premium solution apps (except for an app for Amazon Web Services (AWS)), and several other limitations.

Splunk Free is a free version of the core Splunk Enterprise product that has limits on users (one user), ingestion volume (500 MB/day), and other features; also, it cannot be used for a deployed/clustered configuration.

You can compare the available features of each product here: https://www.splunk.com/en_us/software/features-comparison-chart.html.

The history of Splunk

Splunk was designed from the beginning to allow people who were troubleshooting IT problems to search through their log files as easily as if using a search engine like Google. As a product, Splunk was conceived by founders Rob Das and Erik Swan between 2002 and 2004 after asking a number of people at 60-70 companies, how do you find problems in your infrastructure today? The answer was consistently that they would go through log files, and when asked about what tools they used, they tended to answer that they would write their own scripts and that they do everything manually. They also commented that they used Google to search for posts from other people who had solved the same problem. Rob and Erik thought, why not create a tool that allows people to search through log files as easily as they search the web using something like Google? So they built a prototype to demo at Linux World with the tagline of Google for log files, and since they offered a free download, people tried it, liked it, and told their friends—from there, Splunk spread virally from person to person and company to company.

The name Splunk was derived from asking people, what is it like to troubleshoot in your environment? One answer was that it was like digging through caves with headlamps and helmets on, and crawling through the muck trying to find the problem, and it took forever. A common term for going through caves is spelunking, so they decided to call the product Splunk. From that beginning, Splunk has evolved into the powerful big data and business analytics tool that the founders envisioned from the start.

You can watch Rob and Erik tell their story here: https://www.splunk.com/view/SP-CAAAGBY.

Splunk components

One of the first and most important things you need to learn about Splunk in order to work with it effectively is what the functional components are and how they work together. Here is a list:

Universal forwarder
Indexer and indexer clusters
Search head and search head clusters
Deployment server
Deployer
Cluster master
License master
Heavy forwarder

Universal forwarders, indexers, and search heads constitute the majority of Splunk functionality; the other components provide supporting roles for larger clustered/distributed environments. We'll summarize each of these here and dig into more details in chapters to come.

In very small installations, you can install Splunk Enterprise on a single server, which will provide all the indexing and search functionality in one instance. However, in most cases, but in most cases you will need to scale Splunk up to handle higher data volumes and more users, so the functions that we just listed will be broken out onto multiple servers to support scalability and provide redundancy for higher reliability.

The following diagram shows all the Splunk components involved in a larger Splunk deployment and how they fit together:

Splunk components in a distributed deployment

The universal forwarder (UF) is a free small-footprint version of Splunk Enterprise that is installed on each application, web, or other type of server (which may be running various flavors of Linux or Windows operating systems) to collect data from specified log files and forward this data to Splunk for indexing (storage). In a large Splunk deployment, you may have hundreds or thousands of forwarders that consume and forward data for indexing.

An indexer is the Splunk component that creates and manages indexes, which is where machine data is stored. Indexers perform two main functions: parsing and storing data, which has been received from forwarders or other data sources into indexes, and searching and returning the indexed data in response to search requests.

An indexing cluster is a group of indexers that have been configured to work together to handle higher volumes of both incoming data to be indexed and search requests to be serviced, as well as providing redundancy by keeping duplicate copies of indexed data spread across the cluster members. Be aware that index cluster members can be called both peer nodes and search peers in Splunk documentation, depending on context, which can be a bit confusing until you get used to the nomenclature.

A search head is an instance of Splunk Enterprise that handles search management functions. This includes providing a web-based user interface called Splunk Web, from which users issue search requests in what is called Search Processing Language (SPL). Search requests initiated by a user (or a report or dashboard) are sent to one or more indexers to locate and return the requested data; the search head then formats the returned data for presentation to the user.

A search head cluster is a group of multiple search heads configured to work together to service larger numbers of users and provide redundancy for servicing search requests. Search head cluster members are called cluster members in Splunk documentation.

The following screenshot is an example of executing a simple search in Splunk Web. The SPL specifies searching in the _internal index, which is where Splunk saves data about its internal operations, and provides a count of the number of events in each log for Today. The SPL command specified an index, and then pipes the returned results to the stats command to return a count of all the events by their source and sourcetype (we'll discuss all this a bit further along in this chapter):

index=_internal | stats count by source, sourcetype

Following is the screenshot for simple search in Splunk:

Simple search in Splunk Web

A deployment server is a Splunk Enterprise instance that acts as a centralized configuration manager for a number of Splunk components, but which in practice is used to manage UFs. For instance, all of the hundreds or thousands of UFs that may be installed on servers in an Enterprise environment can periodically "phone home" to the deployment server to pick up new configuration information, making the task of managing all these UFs much easier.

A deployer is a Splunk Enterprise instance that is used to distribute Splunk apps and certain other configuration updates to search head cluster members.

A cluster master is a Splunk Enterprise instance that coordinates the activities of an indexing cluster. In an index cluster, data is distributed across multiple indexer instances to provide redundancy; the cluster master takes care of both controlling how that data is distributed, and helping search heads know where to direct their search requests to find specific sets of data. A cluster master is also called a master node in Splunk documentation.

A license master is a single Splunk Enterprise instance that provides a licensing service for the multiple instances of Splunk that have been deployed in a distributed environment. If you have a standalone instance of Splunk Enterprise, or a standalone indexer, you can install a license locally on that machine; otherwise, you will need a license master.

A heavy forwarder is an instance of Splunk Enterprise that can receive data from other forwarders or data sources and parse, index, and/or send data to another Splunk instance for indexing. A heavy forwarder has some features disabled so that it doesn't consume as many resources as an indexer, but retains most of an indexer's capability except that it cannot service search requests. A heavy forwarder is often used to parse incoming data and route it to various destinations for indexing depending upon the source and type of data received, and/or to speed up processing and reduce the processing load on indexers, but its use is optional.

Splunk Enterprise also has a monitoring tool function called the monitoring console, which lets you view detailed topology and performance information about your entire distributed deployment from one interface. This function can be configured to run on one of the other Splunk components such as the cluster master or deployer if resources allow.

If this seems like a lot of functionality to absorb when you're just getting started, don't worry—you only really need to focus on getting comfortable with the data input, indexing, and search functions for now; the other pieces play smaller roles in day-to-day Splunk administration activities.

Splunk processing tiers

It is also helpful to become familiar with the Splunk processing tiers and what is called the data pipeline, which consists of the transformation functions that occur as data traverses through Splunk software, in order to—better understand how the various components are configured to work together.

As data that Splunk receives from forwarders or other data sources moves through the data pipeline, Splunk transforms it into searchable events that get indexed. The data pipeline has four segments, as illustrated in the following diagram:

Splunk data pipeline

In the Data Input segment, Splunk consumes the raw data stream from forwarders or other data sources, breaks it into 10K blocks, and annotates each block with metadata such as host, source, sourcetype, and the index the data is destined for per the settings in a configuration file.

In the Parsing segment, Splunk breaks data in these blocks into individual events, and identifies, parses, and assigns timestamps to each event. The metadata from the block of data the event came from is assigned to each event, and any required transformations to the event data or metadata are completed.

A Splunk event is a single piece of data in Splunk. Typically, an event represents a single line entry in a log file or data stream, but an event can contain multiple lines such as from a Java stacktrace. We'll cover events in more detail in the next section.

In the Indexing phase, Splunk writes the individual events to the specified index on disk. It writes both compressed raw data and index files; index files contain pointers to the raw data as well as some metadata to facilitate and speed up the search process.

In a distributed deployment, parsing and indexing are usually referenced together as the indexing process, as these functions are both handled on an indexer. This is okay at a high level, but if you need to inspect the processing of data more closely or determine how to scale and allocate Splunk components, you may need to consider the two segments individually.

The Search segment manages all the aspects of how a user searches, views, and uses the indexed data. It parses and interprets the SPL search commands, requests and receives data from indexers, and formats and presents the results to the user. The search function in Splunk also stores and uses user-created knowledge objects such as event types, tags, macros, field extractions, and so on.

In a distributed deployment, the search function is handled on a search head; in a single-instance Splunk Enterprise server, the input, parsing, indexing, and search functions are all accomplished on that single instance, as you might expect. We will cover all these functions, and how they're configured and interact, in much more detail in the next several chapters.

Splunk events

As we discovered in the previous section, Splunk creates events from each entry in a log file or data stream. You can search for specific types of events, within specified time frames, using SPL in Splunk Web. For example, let's say you create a search on the instance of Splunk on your laptop using the SPL command:

index=_internal sourcetype=splunk_web_access

if you press Enter, Splunk will return a number of events for Today or any other time frame you've selected in the Time Range drop-down. These events come from Splunk's internal web server, and reflect the format and fields that are typical of a web log:

Splunk events from a simple search

We'll cover all of the features and details of using Splunk Web in Chapter 6, Searching with Splunk, so for now let's focus on some of the most important and useful fields in the events themselves.

The following is a screenshot of a typical Splunk event:

A typical Splunk event

Regardless of the data source or type, Splunk always tags each event with a number of default fields; some of these come from the metadata mentioned in the discussion about the data pipeline in the previous section, and others are added at index time. There are four of these fields that you will want to become familiar with right away, as they are used extensively for filtering arguments in your SPL commands to return the events of interest. In the preceding screenshot, these key fields have been circled in red – they are as follows:

_time (timestamp)
host
source
sourcetype

The date and time reflected in the Time column is the timestamp assigned to the event, which Splunk stores in a _time field. When an event contains a timestamp, as this one does, that is, [11/Jun/2018:21:12:35.441 -0400], Splunk will parse that timestamp and save it in the _time field as an epoch value (number of seconds since 00:00:00 coordinated universal time (UTC), Thursday, 1 January 1970). If an event does not contain a timestamp, Splunk will assign the time the event was indexed to the _time field. Splunk displays this _time value in the date-time format, as seen previously, corrected for the time zone specified in the Splunk Web account settings—we'll cover this in more detail in the chapter on Splunk search.

The host field is the name or IP address of the physical device from which an event originates. You can use this field to create filters to return events from a specific host. In the preceding example, the host is a Splunk server called robotdev.

The source field identifies where the event originated. In the case of data obtained from log files, the source consists of the full pathname of the file; in the case of a network-based source, this field contains the protocol and port, such as UDP: 514. In this example, the event came from a Splunk log file in the /opt/splunk/var/log/splunk directory called web_access.log.

The sourcetype field identifies the data structure of an event (what fields the event contains, where they are, and how they're formatted), and determines how Splunk parses the data into specified fields during the indexing process. Splunk Enterprise comes with a large set of predefined source types for known data source types, and will assign the correct sourcetype to your data if it recognizes the format. You can use the sourcetype field in searches to find all the data of a certain type, regardless of the source. In the preceding example, the sourcetype is a Splunk-specific type called splunk_web_access.

The other important field that is not displayed in search results, but is essential for writing SPL commands to perform searches is the index, which as you can see was specified in the SPL command used to return the example events previously. Splunk has four internal indexes: _audit, _internal, _introspection, and _telemetry; you can view the data in these to get familiar with events in the short term. You will create and use custom indexes to store data from your company's host and device logs, and specify those indexes in your search strings.

Splunk information resources

The Splunk website has a landing page with links to a vast array of additional information on configuring and using Splunk: https://www.splunk.com/en_us/community.html.

Some of the more useful of these sources include the following:

Splunk documentation is, as you might expect, an extensive collection of documentation and tutorials that covers all aspects of all of Splunk's products. In the Splunk Enterprise documentation landing page, for example, you'll find tabs for Get started, Search and report, Administer, and so on with links to manuals for related topics. The Admin Manual and Getting Data In manuals under the Administer tab are particularly useful for administering Splunk.

Splunk answers is a collection of questions posted by Splunk users and administrators with answers provided by experts in the field who have solved those particular problems. If you use a search engine such as Google to look for an answer to an issue working with Splunk, here or in Splunk Docs is usually where you'll end up.

Splunk education offers a number of Learning Paths for users, Splunk administrators, architects, developers, and so on, which I highly recommend. While this Quick Start Guide is intended to introduce and guide you through the most important and useful features of Splunk from one source, and provide some practical experience and examples of common configurations and challenges you'll encounter, you will want to pursue a formal education path if you wish to obtain a deeper understanding of Splunk and achieve the various levels of Splunk certifications such as Certified User, Power User, Admin, Architect, or Developer.

I also recommend that you download, print, and peruse the Splunk Quick Reference Guide: https://www.splunk.com/pdfs/solution-guides/splunk-quick-reference-guide.pdf.

This is an excellent resource both to accompany the chapters that follow and to keep by your side as you perform day-to-day Splunk administration tasks, as it covers the most significant concepts and information about Splunk as well as a great deal of reference material that is helpful for performing searches using the Search Processing Language.