Using the Universal Forwarder to gather data

Most IT environments today range from multiple servers in the closet of your office to hundreds of endpoint servers located in multiple geographically distributed data centers.

When the data we want to collect is not located directly on the server where Splunk is installed, the Splunk Universal Forwarder (UF) can be installed on your remote endpoint servers and used to forward data back to Splunk to be indexed.

The Universal Forwarder is like the Splunk server in that it has many of the same features, but it does not contain Splunk web and doesn't come bundled with the Python executable and libraries. Additionally, the Universal Forwarder cannot process data in advance, such as performing line breaking and timestamp extraction.

This recipe will guide you through configuring the Splunk Universal Forwarder to forward data to a Splunk indexer and will show you how to set up the indexer to receive the data.

Getting ready

To step through this recipe, you will need a server with the Splunk Universal Forwarder installed but not configured. You will also need a running Splunk Enterprise server. No other prerequisites are required.

To obtain the Universal Forwarder software, you need to go to https://www.splunk.com/download and register for an account if you do not already have one. Then, either download the software directly to your server or download it to your laptop or workstation and upload it to your server using a file transfer process such as SFTP.
For more information on how to install and manage the Universal Forwarder, visit https://docs.splunk.com/Documentation/Forwarder/latest/Forwarder/Abouttheuniversalforwarder.

How to do it...

Follow these steps to configure the Splunk Forwarder to forward data and the Splunk indexer to receive data:

1. On the server with the Universal Forwarder installed, open a command prompt if you are a Windows user or a terminal window if you are a Unix user.
2. Change to the $SPLUNK_HOME/bin directory, where $SPLUNK_HOME is the directory in which the Splunk forwarder was installed.
3. For Unix, the default installation directory will be /opt/splunkforwarder/bin. For Windows, it will be C:/Program Files/SplunkUniversalForwarder/bin.

If using Windows, omit ./ in front of the Splunk command in the upcoming steps.

4. Start the Splunk forwarder, if not already started, using the following command:

./splunk start   

5. Accept the license agreement.
6. Enable the Universal Forwarder to autostart, using the following command:

./splunk enable boot-start   

7. Set the indexer that this Universal Forwarder will send its data to. Replace the host value with the value of the indexer as well as the username and password for the Universal Forwarder, using the following command:

    ./splunk add forward-server <host>:9997 -auth <username>:<password>  

8. The username and password to log in to the forwarder (default is admin:changeme) is <username>:<password>.

Additional receiving indexers can be added in the same way by repeating the command in the previous step with a different indexer host or IP. Splunk will automatically load balance the forwarded data if more than one receiving indexer is specified in this manner. Port 9997 is the default Splunk TCP port and should only be changed if it cannot be used for some reason.

9. On the receiving Splunk indexer servers, log in to your receiving Splunk indexer server. From the home launcher, in the top right-hand corner, click on the Settings menu item and then select the Forwarding and receiving link:
   
10. Click on the Configure receiving link:
    
11. Click on New.
12. Enter 9997 in the Listen on this port field:
    

13. Click on Save and restart Splunk. The Universal Forwarder is installed and configured to send data to your Splunk server, and the Splunk server is configured to receive data on the default Splunk TCP port 9997.

How it works...

When you tell the forwarder which server to send data to, you basically add a new configuration stanza into an outputs.conf file behind the scenes. On the Splunk server, an inputs.conf file will contain a [splunktcp] stanza to enable receiving. The outputs.conf file on the Splunk forwarder will be located in $SPLUNK_HOME/etc/system/local, and the inputs.conf file on the Splunk server will be located in the local directory of the app you were in (the launcher app in this case) when configuring receiving.

Using forwarders to collect and forward data has many advantages. The forwarders communicate with the indexers on TCP port 9997 by default, which makes for a very simple set of firewall rules that need to be opened. Forwarders can also be configured to load balance their data across multiple indexers, increasing search speeds and availability. Additionally, forwarders can be configured to queue the data they collect if communication with the indexers is lost. This can be extremely important when collecting data that is not read from logfiles, such as performance counters or syslog streams, as the data cannot be re-read.

There's more...

While configuring the settings of the Universal Forwarder can be performed using the command-line interface of Splunk, as outlined in this recipe, there are several other methods to update the settings quickly and to allow for customization of the many configuration options that Splunk provides.

Adding the receiving indexer via outputs.conf

The receiving indexers can be directly added to the outputs.conf configuration file on the Universal Forwarder. Edit $SPLUNK_HOME/etc/system/local/outputs.conf, add your input, and then restart the UF. The following example configuration is provided, where two receiving indexers are specified. The [tcpout-server] stanza can be leveraged to add output configurations specific to an individual receiving indexer:

[tcpout] 
defaultGroup = default-autolb-group 
 
[tcpout:default-autolb-group] 
disabled = false 
server = mysplunkindexer1:9997,mysplunkindexer2:9997 
 
[tcpout-server://mysplunkindexer1:9997] 
[tcpout-server://mysplunkindexer2:9997] 

If nothing has been configured in inputs.conf on the UF, but outputs.conf is configured with at least one valid receiving indexer, the Splunk forwarder will only send internal forwarder health-related data to the indexer. It is therefore possible to configure a forwarder correctly and the forwarder be detected by the Splunk indexers, but not actually send any real data.