Also, if you are a UI engineer using Java or .NET or AJAX (jQuery, ExtJS, etc..) technologies, and are motivated to move to ActionScript/Flex, we will provide you with the tools and mentoring to be successful in this position.

Experience in building network topology visualizations is a big plus!

All resumes can be emailed to me directly.

Serveral customers have reported that DeploymentServer configuration bundles were not taking effect, only to realize after several troubleshooting cycles that there was some configuration in /etc/system/local that was preventing that from happening. Note that any configuration in /etc/system/local will always take precedence over any other configuration in the system - even deployed bundles.

So, if you are stuck in this position, please make sure to check your /etc/system/local before hitting the panic button!

Forwarding metrics.log

Forwarding metrics.log will require that you make the following changes to the configuration on each Splunk instance that you would like to collect the metrics from:

If you have many Splunks in your environment, then making these changes on each one of them manually is certainly not an option you would cherish. This is where Deployment Server can help you centralize all your configurations in one place and distribute them to all or selected instances.

Here’s something I like to do

1. Have all Splunks point to a common Deployment Server

This can be achieved very easily by creating/editing deployment.conf in $SPLUNK_HOME/etc/system/local on each Splunk instance.

[deployment-client]
deploymentServerUri=<your_deployment_server_uri>:<mgmt_port>

For some of my distributed testing on EC2, I have images that include this configuration in the default image (AMI). Using this approach guarantees that configurations never ever have to be changed by hand!

2. Create a bundle

Create a bundle by any name (I called it deployable) and make sure it is available in your Deployment Server’s serverClassPath. This bundle should have two files - inputs.conf and outputs.conf - as described above - here’s a sample bundle you could re-use.

3. Make the bundle available to all Splunks

Make all deployment clients that connect to the deployment server to be part of the deployable service class. This is achieved by changing deployment.conf on Deployment Server again as:

[distributedDeployment-classMaps]
*=deployable

4. Refresh Deployment Server Configuration

This CLI on your Deployment Server instance will make it aware of the new configuration without a restart:

splunk reload deploy-server -auth admin:changeme

You are now all set and all Splunks in your environment will automagically download and apply the bundles within a minute! And in another 30 seconds, your Deployment Server will start aggregating metrics information about your entire data-center!

We want to hear about your experiences in managing Splunk - use the Comments below or send me an email directly at inder@splunk.com.

Forwarding-side

Splunk uses a component called TcpOutputProcessor, which is configured using outputs.conf, to forward data to another Splunk or non-Splunk entity. This is something that a lot of people also refers to as a forwarder. Each TcpOutputProcessor instance publishes metrics events every 30 seconds - all the fields of these events are described below:

• group=tcpout_connections - this field discriminates this event as being a TcpOutput metric.
• tcpout_group_name:destIp:destPort - the load-balanced group that this metric belongs to. If you have multiple groups defined, a separate event is published for each of those groups.
• host metadata - is always available in an event, and refers to the host on which the forwarder is running.
• sourcePort - the local port that is used to connect to the remote entity.
• destIp - the ip address of the remote server to which events are being forwarded.
• destPort - the destination port on which events are being forwarded.
• tcp_bps - bytes per second averaged over last 30 seconds.
• tcp_kbprocessed - total KBytes processed since this connection went live.
• tcp_eps - events per second averaged over last 30 seconds.
• tcp_dropped_events - number of events dropped on this connection.

Indexing side

Similarly on the indexing side, if you have configured inputs.conf to receive data from one or more forwarders, a metrics event is published every 30 seconds for each connection into your indexer. All the fields of a metrics event on the input side are described below:

• group=tcpin_connections - this field discriminates this event as being an input metric.
• sourceHost - The hostname of the entity that is forwarding data to this indexer. If hostname is not available, then it’s IP address is used.
• sourcePort - The remote port of the forwarding entity.
• destPort - The local port on the input side for which this metric is being collected. Typically this port is defined in inputs.conf.
• tcp_bps - bytes per second averages over last 30 seconds.
• tcp_kprocessed - KBytes processed since the connection was established.
• tcp_eps - Events per second averaged over 30 seconds.

These metrics will now enable you to get unusual insight into the operation of your forwarders and indexers. Here’s a sample query that you can run on each indexer instance to get a report on thruput by each forwarding entity:

index=_internal metrics "group=tcpin_connections" | timechart span=30s avg(tcp_bps) by sourceHost

Also, I created a saved search, and used Splunk’s reporting features to always show me the current status on a dashboard.

Now that you have all of this nice data, I am sure you would like it all aggregated in one location.

Good luck playing with these metrics, and if you have any suggestions on what more you would like to see, drop me a line at inder@splunk.com.