Data onboarding – defining field extractions

Splunk has many built-in features, including knowledge of several common source types, which lets it automatically know which fields exist within your data. Splunk, by default, also extracts any key-value pairs present within the log data and all the fields within JSON-formatted logs. However, often the fields within raw log data cannot be interpreted out of the box, and this knowledge must be provided to Splunk to make these fields easily searchable.

The sample data that we will be using in subsequent chapters contains data we wish to present as fields to Splunk. Much of the raw log data contains key-value fields that Splunk will extract automatically, but there is one field we need to tell Splunk how to extract, representing the page response time. To do this, we will be adding a custom field extraction, which will tell Splunk how to extract the field for us.

Getting ready

To step through this recipe, you will need a running Splunk server with the operational intelligence sample data loaded. No other prerequisites are required.

How to do it...

Follow these steps to add a custom field extraction for a response:

1. Log in to your Splunk server.
2. In the top right-hand corner, click on the Settings menu and then click on the Fields link.

3. Click on the Field extractions link:
   
4. Click on New.
5. In the Destination app field, select the search app, and in the Name field, enter response. Set the Apply to dropdown to sourcetype and the named field to access_combined. Set the Type dropdown to Inline, and for the Extraction/Transform field, carefully enter the (?i)^(?:[^"]*"){8}s+(?P<response>.+) regex:
   
6. Click on Save.
7. On the Field extractions listing page, find the recently added extraction, and in the Sharing column, click on the Permissions link:
   

8. Update the Object should appear in setting to All apps. In the Permissions section, for the Read column, check Everyone, and in the Write column, check admin. Then, click on Save:
   
9. Navigate to the Splunk search screen and enter the following search over the Last 60 minutes time range:

index=main sourcetype=access_combined 

10. You should now see a field called response extracted on the left-hand side of the search screen under the Interesting Fields section.

How it works...

All field extractions are maintained in the props.conf and transforms.conf configuration files. The stanzas in props.conf include an extraction class that leverages regular expressions to extract field names and/or values to be used at search time. The transforms.conf file goes further and can be leveraged for more advanced extractions, such as reusing or sharing extractions over multiple sources, source types, or hosts.

See also

• The Loading the sample data for this book recipe
• The Data onboarding – defining event types and tags recipe