Deployed bundles not taking effect?

Changes made in /etc/system/local override any configuration bundles that you may be trying to publish to your Splunk instances using a DeploymentServer.

Serveral customers have reported that DeploymentServer configuration bundles were not taking effect, only to realize after several troubleshooting cycles that there was some configuration in /etc/system/local that was preventing that from happening. Note that any configuration in /etc/system/local will always take precedence over any other configuration in the system – even deployed bundles.

So, if you are stuck in this position, please make sure to check your /etc/system/local before hitting the panic button!…

» Continue reading

Aggregating Metrics from all your Splunks…

If you found that the new metrics being generated by Splunk on the input (indexing in many cases) and forwarding side to be useful, I am sure you would want to aggregate them all in a central location. Well, you can do that by using Splunk’s forwarding mechanism itself! Although, it does not matter where you aggregate these metrics, I believe the Deployment Server instance could be a good location, if you have one setup for your installation.

Forwarding metrics.log

Forwarding metrics.log will require that you make the following changes to the configuration on each Splunk instance that you would like to collect the metrics from:

  • Edit or create inputs.conf in $SPLUNK_HOME/etc/system/local folder

    [monitor://$SPLUNK_HOME/var/log/splunk/metrics.log]

    _TCP_ROUTING = RouteMetricsToDeploymentServer

  • Similarly for outputs.conf
  • » Continue reading

    Forwarder and Indexer Metrics

    If you were always wondering how much data was being transferred between your forwarders and indexers, we may have some help for you. Splunk now publishes these metrics to metrics.log, which are by default tailed and indexed in “_internal”.

    Forwarding-side

    Splunk uses a component called TcpOutputProcessor, which is configured using outputs.conf, to forward data to another Splunk or non-Splunk entity. This is something that a lot of people also refers to as a forwarder. Each TcpOutputProcessor instance publishes metrics events every 30 seconds – all the fields of these events are described below:

    • group=tcpout_connections – this field discriminates this event as being a TcpOutput metric.
    • tcpout_group_name:destIp:destPort – the load-balanced group that this metric belongs to. If you have
    » Continue reading