Using CSV files to store transient data

Sometimes it is useful to store small amounts of data outside of a Splunk index. Using the inputcsv and outputcsv commands, we can store tabular data in CSV files on the filesystem.

Pre-populating a dropdown

If a dashboard contains a dynamic dropdown, you must use a search to populate the dropdown. As the amount of data increases, the query to populate the dropdown will run more and more slowly, even from a summary index. We can use a CSV file to store just the information needed, simply adding new values when they occur.

First, we build a query to generate the CSV file. This query should be run over as much data as possible:

source="impl_splunk_gen"
  | stats count by user
  | outputcsv user_list.csv

Next, we need a query to run periodically that will append any new entries to the file. Schedule this query to run periodically as a saved search:

source="impl_splunk_gen"
  | stats count by user
  | append [inputcsv user_list.csv]
  | stats sum(count) as count...