If you found that the new metrics being generated by Splunk on the input (indexing in many cases) and forwarding side to be useful, I am sure you would want to aggregate them all in a central location. Well, you can do that by using Splunk’s forwarding mechanism itself! Although, it does not matter where you aggregate these metrics, I believe the Deployment Server instance could be a good location, if you have one setup for your installation.

Forwarding metrics.log

Forwarding metrics.log will require that you make the following changes to the configuration on each Splunk instance that you would like to collect the metrics from:

If you have many Splunks in your environment, then making these changes on each one of them manually is certainly not an option you would cherish. This is where Deployment Server can help you centralize all your configurations in one place and distribute them to all or selected instances.

Here’s something I like to do

1. Have all Splunks point to a common Deployment Server

This can be achieved very easily by creating/editing deployment.conf in $SPLUNK_HOME/etc/system/local on each Splunk instance.

If you were always wondering how much data was being transferred between your forwarders and indexers, we may have some help for you. Splunk now publishes these metrics to metrics.log, which are by default tailed and indexed in “_internal”.

Forwarding-side

Splunk uses a component called TcpOutputProcessor, which is configured using outputs.conf, to forward data to another Splunk or non-Splunk entity. This is something that a lot of people also refers to as a forwarder. Each TcpOutputProcessor instance publishes metrics events every 30 seconds - all the fields of these events are described below:

• group=tcpout_connections - this field discriminates this event as being a TcpOutput metric.
• tcpout_group_name:destIp:destPort - the load-balanced group that this metric belongs to. If you have multiple groups defined, a separate event is published for each of those groups.
• host metadata - is always available in an event, and refers to the host on which the forwarder is running.
• sourcePort - the local port that is used to connect to the remote entity.
• destIp - the ip address of the remote server to which events are being forwarded.
• destPort - the destination port on which events are being forwarded.