Splunk is hiring ActionScript/Flex engineers to build new products for the Enterprise team. If you have been building Enterprise and/or Web applications using AS/Flex, we would love to talk to you.

Also, if you are a UI engineer using Java or .NET or AJAX (jQuery, ExtJS, etc..) technologies, and are motivated to move to ActionScript/Flex, we will provide you with the tools and mentoring to be successful in this position.

Experience in building network topology visualizations is a big plus!

All resumes can be emailed to me directly.

Changes made in /etc/system/local override any configuration bundles that you may be trying to publish to your Splunk instances using a DeploymentServer.

Serveral customers have reported that DeploymentServer configuration bundles were not taking effect, only to realize after several troubleshooting cycles that there was some configuration in /etc/system/local that was preventing that from happening. Note that any configuration in /etc/system/local will always take precedence over any other configuration in the system - even deployed bundles.

So, if you are stuck in this position, please make sure to check your /etc/system/local before hitting the panic button!

If you found that the new metrics being generated by Splunk on the input (indexing in many cases) and forwarding side to be useful, I am sure you would want to aggregate them all in a central location. Well, you can do that by using Splunk’s forwarding mechanism itself! Although, it does not matter where you aggregate these metrics, I believe the Deployment Server instance could be a good location, if you have one setup for your installation.

Forwarding metrics.log

Forwarding metrics.log will require that you make the following changes to the configuration on each Splunk instance that you would like to collect the metrics from:

If you have many Splunks in your environment, then making these changes on each one of them manually is certainly not an option you would cherish. This is where Deployment Server can help you centralize all your configurations in one place and distribute them to all or selected instances.

Here’s something I like to do

1. Have all Splunks point to a common Deployment Server

This can be achieved very easily by creating/editing deployment.conf in $SPLUNK_HOME/etc/system/local on each Splunk instance.

If you were always wondering how much data was being transferred between your forwarders and indexers, we may have some help for you. Splunk now publishes these metrics to metrics.log, which are by default tailed and indexed in “_internal”.

Forwarding-side

Splunk uses a component called TcpOutputProcessor, which is configured using outputs.conf, to forward data to another Splunk or non-Splunk entity. This is something that a lot of people also refers to as a forwarder. Each TcpOutputProcessor instance publishes metrics events every 30 seconds - all the fields of these events are described below:

• group=tcpout_connections - this field discriminates this event as being a TcpOutput metric.
• tcpout_group_name:destIp:destPort - the load-balanced group that this metric belongs to. If you have multiple groups defined, a separate event is published for each of those groups.
• host metadata - is always available in an event, and refers to the host on which the forwarder is running.
• sourcePort - the local port that is used to connect to the remote entity.
• destIp - the ip address of the remote server to which events are being forwarded.
• destPort - the destination port on which events are being forwarded.