If you found that the new metrics being generated by Splunk on the input (indexing in many cases) and forwarding side to be useful, I am sure you would want to aggregate them all in a central location. Well, you can do that by using Splunk’s forwarding mechanism itself! Although, it does not matter where you aggregate these metrics, I believe the Deployment Server instance could be a good location, if you have one setup for your installation.

Forwarding metrics.log

Forwarding metrics.log will require that you make the following changes to the configuration on each Splunk instance that you would like to collect the metrics from:

If you have many Splunks in your environment, then making these changes on each one of them manually is certainly not an option you would cherish. This is where Deployment Server can help you centralize all your configurations in one place and distribute them to all or selected instances.

Here’s something I like to do

1. Have all Splunks point to a common Deployment Server

This can be achieved very easily by creating/editing deployment.conf in $SPLUNK_HOME/etc/system/local on each Splunk instance.

[deployment-client]
deploymentServerUri=<your_deployment_server_uri>:<mgmt_port>

For some of my distributed testing on EC2, I have images that include this configuration in the default image (AMI). Using this approach guarantees that configurations never ever have to be changed by hand!

2. Create a bundle

Create a bundle by any name (I called it deployable) and make sure it is available in your Deployment Server’s serverClassPath. This bundle should have two files - inputs.conf and outputs.conf - as described above - here’s a sample bundle you could re-use.

3. Make the bundle available to all Splunks

Make all deployment clients that connect to the deployment server to be part of the deployable service class. This is achieved by changing deployment.conf on Deployment Server again as:

[distributedDeployment-classMaps]
*=deployable

4. Refresh Deployment Server Configuration

This CLI on your Deployment Server instance will make it aware of the new configuration without a restart:

splunk reload deploy-server -auth admin:changeme

You are now all set and all Splunks in your environment will automagically download and apply the bundles within a minute! And in another 30 seconds, your Deployment Server will start aggregating metrics information about your entire data-center!

We want to hear about your experiences in managing Splunk - use the Comments below or send me an email directly at inder@splunk.com.