In the previous chapter, we learned about the various ways to get data into Splunk. In this chapter, we will dive right into the data and get our hands dirty.

The ability to search machine data is one of Splunk's core functions, and it should come as no surprise that many other features and functions of Splunk are heavily driven by searches. Everything from basic reports and dashboards through to data models and fully featured Splunk applications is powered by Splunk searches behind the scenes.

Splunk has its own search language known as the Search Processing Language (SPL). This SPL contains hundreds of search commands, most of which also have several functions, arguments...

When a basic search is executed in Splunk from the search bar, the search results are displayed in a raw event format by default. To many users, this raw event information is not particularly readable, and valuable information is often clouded by other less valuable data within the event. Additionally, if events span several lines, only a few events can be seen on screen at any one time.

In this recipe, we will write a Splunk search to demonstrate how we can leverage Splunk commands to make raw event data readable, tabulating events and displaying only the fields we are interested in.

To step through this recipe, you will need a running Splunk Enterprise server, with the sample...

One of the data samples we loaded in Chapter 1, Play Time - Getting Data In, contained access logs from our web server. These have a Splunk source type of access_combined and detail all pages accessed by users of our web application. We are particularly interested in knowing which pages are being accessed the most, as this information provides great insight into how our e-commerce web application is being used. It could also help influence changes to our web application such that rarely visited pages are removed, or our application is redesigned to be more efficient.

In this recipe, we will write a Splunk search to find the most accessed web pages over a given period of time.

Users visiting our website use a variety of devices and web browsers. By analyzing the web access logs, we can understand which browsers are the most popular and therefore which browsers our site must support at the least. We can also use this same information to help identify the types of devices that people are using.

In this recipe, we will write a Splunk search to find the most used web browsers over a given period of time. We will then make use of both the eval and replace commands to clean up the data a bit.

To step through this recipe, you will need a running Splunk Enterprise server with the sample data loaded from Chapter 1, Play Time - Getting Data In. You should be...

Our web access logs continue to give us great information about our website and the users visiting the site. Understanding where our users are coming from provides insight into potential sales leads and/or which marketing activities might be working better than others. For this information, we look for the referer_domain field value within the log data.

In this recipe, we will write a Splunk search to find the top-referring websites.

To step through this recipe, you will need a running Splunk Enterprise server, with the sample data loaded from Chapter 1, Play Time – Getting Data In. You should be familiar with the Splunk search bar and the time range picker.

Log data often contains seemingly cryptic codes that have all sorts of meanings. This is true of our web access logs, where there is a status code that represents a web page response. This code is very useful, as it can tell us whether certain events were successful or not. For example, error codes found in purchase events are less than ideal, and if our website was at fault, then we might have lost a sale.

In this recipe, we will write a Splunk search to chart web page responses against the various web pages on the site.

To step through this recipe, you will need a running Splunk Enterprise server, with the sample data loaded from Chapter 1, Play Time - Getting Data In. You should...

No one likes to wait for a web page to load, and we certainly do not want users of our web application waiting either! Within our web access logs, there is a field named response that tracks the total time the page has taken to load in milliseconds.

In this recipe, we will track the average page load time over the past week at different times of the day.

To step through this recipe, you will need a running Splunk Enterprise server, with the sample data loaded from Chapter 1, Play Time – Getting Data In. You should be familiar with the Splunk search bar and the time range picker.

Our web access logs capture the product IDs (the item field in the logs) that users are viewing and adding to their shopping carts most often. Understanding the top products that people view can help influence our sales and marketing strategy, and even product direction. Additionally, products viewed on an e-commerce website might not always necessarily translate into sales of that product.

In this next recipe, we will write a Splunk search to chart the 10 ten products that users successfully view and compare against the number of successful shopping cart additions for each product. For example, if a product has a high number of views but is not being added to carts and is subsequently purchased, this could indicate that something is not right; perhaps the pricing of the product is too high.

Another of the data samples we loaded in Chapter 1, Play Time - Getting Data In, contained application logs from our application server. These have a Splunk source type of log4j and detail the various calls that our application makes to the backend database in response to user web requests, in addition to providing insight into memory utilization and other health-related information. We are particularly interested in tracking how our application is performing in relation to the time taken to process user-driven requests for information.

In this recipe, we will write a Splunk search to find out how our application is performing. To do this, we will analyze database call transactions and chart the maximum, mean, and minimum transaction durations over the past week.

In addition to measuring the functional performance of database transactions, we are also interested in understanding how our application is performing from a memory usage perspective. Analyzing this type of information can help identify memory leaks in our application or high memory utilization that might be affecting the user experience and causing our application to slow down.

In this recipe, we will analyze the memory usage of our application over time.

To step through this recipe, you will need a running Splunk Enterprise server, with the sample data loaded from Chapter 1, Play Time – Getting Data In. You should be familiar with the Splunk search bar and...

Our application currently only allows for a limited number of concurrent database connections. As our application user base grows, we need to monitor these connections proactively to ensure that we do not hit our concurrency limit and we know when we need to scale out the database infrastructure further.

In the last recipe of this chapter, we will monitor database transactions over the past week to identify if there are certain times or days when we might be close to our concurrency limit.

To step through this recipe, you will need a running Splunk Enterprise server, with the sample data loaded from Chapter 1, Play Time - Getting Data In. You should be familiar...