First and foremost, try to determine the scope of your App. Once you have the scope planned out, try to limit the amount of scope creep that occurs, if possible. You may just be trying to perform extractions on your data, and if that is your current end goal, stop there. Don't try to build a full-blown suite on your first attempt. Build the IA, then the TA, and then move on from there. Ask yourself these questions as you try to determine your scope:

These questions can help you spark an idea of what internal resources would need to be engaged, as well as any kind of documentation and educational requirements.

Once you have determined the scope of the App, you will need to decide how and from where you will consume the data. Getting data into Splunk can happen in a very wide variety of ways. There is no set manner of input that will work on all data sources. You may have to develop a new script or modular input. Being aware of where your data is coming from is the key to getting it consumed correctly the first time. A few questions you may ask yourself could be:

There is a lot of data out in the wild, but not all of it may be relevant to your use case. You may find that of a service that has 100 endpoints available for data collection, you only need 10. Not only will you save on license usage, but your indexers will thank you for it as well.

Another key thought process in App development is how far you want to brand your App. Splunk has a very robust architecture and framework, providing you with the ability to customize your Apps extensively. You can override any individual piece of CSS and extend SplunkJS Stack to include any number of different visualizations or third-party libraries. Additional questions you might ponder on would include:

There are so many options available to brand your App, but all customizations should conform to the Splunk branding guidelines for developers. Visit http://www.splunk.com/view/SP-CAAAFT9 to read through Splunk's guidelines.

Once you have the whats and hows of the data you're going to collect, you need to figure out visualizations. How you display the information is just as important as what data you collect. Splunk comes with a variety of graphs and displays right out of the box, and can be extended quite easily to include some really cool presentations. Some of the questions posed to you might be:

Such a small list of questions hardly precludes any other relevant discussion within your organization. The more internal discussion that can take place, the better and more thought-out your App may turn out.

Installing Apps via Splunk Web is simple. Once you have downloaded the App from its source, you navigate to the Manage Apps section of Splunk. You will find this at the top-left of Splunk Web, as shown in the following screenshot:

Once you have clicked on Manage Apps, you will see a button to install the app from a file. You can also browse the Splunk App store, using the first button:

This brings you to a form that you can use to actually install the App. Simply click on the Browse button, select the file you downloaded, check the Upgrade button if this App has already been installed, and then click on Upload. That's it! Splunk takes the App, installs it, and prompts to restart if needed:

CLI holds a special place in many *nix admins' hearts. It is entirely possible to install Apps via the command line alone. Doing so requires having the following: access to the physical (or virtual) server and enough permissions to perform CLI commands with Splunk. All commands are going to be executed from $SPLUNK_HOME, which normally defaults to /opt/splunk. Follow these steps to install an App via CLI:

Splunk will install the App. You may be prompted to restart, depending on the contents of the App. Index-time configurations require a restart, whereas search-time configurations do not.