The key to creating an effective search is to take advantage of the index. The Splunk index is effectively a huge word index, sliced by time. One of the most important factors for the performance of your searches is how many events are pulled from the disk. The following few key points should be committed to memory:

• Search terms are case insensitive: Searches for error, Error, ERROR, and ErRoR are all the same.
• Search terms are additive: Given the search item mary error, only events that contain both words will be found. There are Boolean and grouping operators to change this behavior; we will discuss these later.
• Only the time frame specified is queried: This may seem obvious, but it's very different from a database, which would always have a single index across all events in a table. Since each index is sliced into new buckets over time...

There are a few operators that you can use to refine your searches (note that these operators must be in uppercase so as not to be considered search terms):

• AND is implied between terms. For instance, error mary (two words separated by a space) is the same as error AND mary.
• OR allows you to specify multiple values. For instance, error OR mary means find any event that contains either word.
• NOT applies to the next term or group. For example, error NOT mary would find events that contain error but do not contain mary.
• The quote marks ("") identify a phrase. For example, "Out of this world" will find this exact sequence of words. Out of this world will find any event that contains all of these words, but not necessarily in that order.
• Parentheses ( ( ) ) are used for grouping terms. Parentheses can help avoid confusion...

Though you can probably figure it out by just clicking around, it is worth discussing the behavior of the GUI when moving your mouse around and clicking:

• Clicking on any word or field value will give you the option to Add to search or Exclude from search the existing search or create a New search, as shown in the following screenshot:

• Clicking on a word or a field value that is already in the query will give you the option to remove it from the existing query or, as previously, create a new search, as shown in the following screenshot:

In prior versions of Splunk, event segmentation was configurable through a setting in the Options dialog...

When we explored the GUI in Chapter 1, The Splunk Interface, you probably noticed fields everywhere. Fields appear in the field picker on the left and under every event. Where fields actually come from is transparent to the user, who simply searches for key=value. We will discuss adding new fields in Chapter 3, Tables, Charts, and Fields, and in Chapter 11, Configuring Splunk.

The field picker gives us easy access to the fields (currently defined) for the results of our query. Splunk will extract some fields from event data without your help such as: host, source, and sourcetype values, timestamps, and others. Additional fields to be extracted can be defined by you. Clicking on...

Though the index is based on words, it is possible to use wildcards when necessary, albeit a little carefully. Take a look at some interesting facts about wildcards:

• Only trailing wildcards are efficient: Stated simply, bob* will find events containing Bobby efficiently, but *by or *ob* will not. The latter cases will scan all events in the time frame specified.
• Wildcards are tested last: Wildcards are tested after all other terms. Given the search authclass *ob* hello world, all other terms besides *ob* will be searched first. The more you can limit the results using full words and fields, the better your search will perform.

Given the following events,...

Time is an important and confusing topic in Splunk. If you want to skip this section, absorb one concept: time must be parsed properly on the way into the index, as it cannot be changed later without indexing the raw data again.

Given the date 11-03-04, how would you interpret this date? Your answer probably depends on where you live. In the United States, you would probably read this as November 3, 2004. In Europe, you would probably read this as March 11, 2004. It would also be reasonable to read this as March 4, 2011.

Luckily, most dates are not this ambiguous, and Splunk makes a good effort to find and extract them, but it is absolutely worth the trouble to give Splunk a little help...

We have talked about using the index to make searches faster. When starting a new investigation, following a few steps will help you get results faster:

1. Set the time to the minimum time that you believe will be required to locate relevant events. For a chatty log, this may be as little as a minute. If you don't know when the events occurred, you might search a larger time frame and then zoom in by clicking on the timeline while the search is running.
2. Specify the index if you have multiple indexes. It's good to get into the habit of starting your queries with the index name. For example, index=myapplicationindex error bob.
3. Specify other fields that are relevant. The most common fields to specify are sourcetype and host. For example, index=myapplicationindex sourcetype="impl_splunk_gen" error bob. If you find yourself specifying the field...