In this section, I'll provide a few searches that are helpful for interrogating some performance and capacity aspects of your Splunk deployment; these are in addition to a number of useful displays you can get from the Monitoring Console, which we'll cover in the 'Splunk monitoring console' section of this chapter shortly:
- How quickly are indexers responding to search heads? (run on a search head) This could identify a search that is poorly written. The results indicate the search ID (search_id), and you can match the values in the results to directory names in the $SPLUNK_HOME/var/run/splunk/dispatch directory if you want to inspect the search artifacts. Here's the search:
index=_internal source=*remote_searches.log
| stats max(elapsedTime) as MaxElapsedTime by server, search_id
| convert num(MaxElapsedTime) | where MaxElapsedTime...
