Digital Resilience Pays Off
Download this e-book to learn about the role of Digital Resilience across enterprises.
If you found that the new metrics being generated by Splunk on the input (indexing in many cases) and forwarding side to be useful, I am sure you would want to aggregate them all in a central location. Well, you can do that by using Splunk’s forwarding mechanism itself! Although, it does not matter where you aggregate these metrics, I believe the Deployment Server instance could be a good location, if you have one setup for your installation.
Forwarding metrics.log will require that you make the following changes to the configuration on each Splunk instance that you would like to collect the metrics from:
inputs.conf
in $SPLUNK_HOME/etc/system/local
folder[monitor://$SPLUNK_HOME/var/log/splunk/metrics.log]
_TCP_ROUTING = RouteMetricsToDeploymentServer
outputs.conf
[tcpout]
disabled=false
[tcpout:RouteMetricsToDeploymentServer]
server=<deployment_sever_ip>:<deployment_server_port>
If you have many Splunks in your environment, then making these changes on each one of them manually is certainly not an option you would cherish. This is where Deployment Server can help you centralize all your configurations in one place and distribute them to all or selected instances.
Here’s something I like to do
This can be achieved very easily by creating/editing deployment.conf
in $SPLUNK_HOME/etc/system/local
on each Splunk instance.
[deployment-client]
deploymentServerUri=<your_deployment_server_uri>:<mgmt_port>
For some of my distributed testing on EC2, I have images that include this configuration in the default image (AMI). Using this approach guarantees that configurations never ever have to be changed by hand!
Create a bundle by any name (I called it deployable) and make sure it is available in your Deployment Server’s serverClassPath
. This bundle should have two files – inputs.conf and outputs.conf – as described above – here’s a sample bundle you could re-use.
Make all deployment clients that connect to the deployment server to be part of the deployable service class. This is achieved by changing deployment.conf on Deployment Server again as:
[distributedDeployment-classMaps]
*=deployable
This CLI on your Deployment Server instance will make it aware of the new configuration without a restart:
splunk reload deploy-server -auth admin:changeme
You are now all set and all Splunks in your environment will automagically download and apply the bundles within a minute! And in another 30 seconds, your Deployment Server will start aggregating metrics information about your entire data-center!
We want to hear about your experiences in managing Splunk – use the Comments below or send me an email directly at inder@splunk.com.
----------------------------------------------------
Thanks!
Inder Sabharwal
The Splunk platform removes the barriers between data and action, empowering observability, IT and security teams to ensure their organizations are secure, resilient and innovative.
Founded in 2003, Splunk is a global company — with over 7,500 employees, Splunkers have received over 1,020 patents to date and availability in 21 regions around the world — and offers an open, extensible data platform that supports shared data across any environment so that all teams in an organization can get end-to-end visibility, with context, for every interaction and business process. Build a strong data foundation with Splunk.