If you are a long time enterprise user of the 3.x product, you may have become used to the pull-down menu for distributed searching.   One of the common use cases for this menu was searching specific indexers in your distributed search.   A common question was:  “Can we restrict the server via search syntax?”.   In the 3.3 and 3.4 product, you cannot restrict via syntax through the web interface.   There is a trick you can use via the command line, but that doesn’t help when you want to do this in a saved search.

In the 4.0 release, we have removed the pull-down menu and implemented indexer restrictions with search syntax. The new parameter is called “splunk_server”.   Let’s assume I have a distributed searcher (hostname=searcher1) and three indexers (hostname=indexer1, hostname=indexer2, and hostname=indexer3).  If I am searching for “error” and my goal is to restrict my searches to indexer3, I would use the following query:

splunk_server=indexer3 error

To search anything but indexer3 I would use:

error NOT splunk_server=indexer3

Using this restriction can be useful for tracking specific datacenters, monitoring server health, and securing data (can add this as a filter to a role).  For the complete documentation on this command, see our official documentation:

http://www.splunk.com/base/Documentation/latest/User/SpecifyMultipleServersToSearch

Note:   distributed searching is limited to the Splunk enterprise version.

There are many ways to feed data into Splunk. One method is to monitor the files within a directory. In the default ‘monitor’ configuration, Splunk will try to index all files within a specified directory. In some cases, you may have a directory which contains many files including some that you do not want to index. Splunk can be configured to index specific file types as well as sub directories. Here is a real-world working example of how to use a white list…

Let us assume we want to index certain compressed files (*.gz) where the file name starts with “200906″. One of the filename’s is “20090631.gz”. These files exist in a specific directory: “/storage/datacenter/host1/webserver”.  To make things more interesting, I have other *.log files in that directory. There are also other subdirectories within datacenter (such as host2, router1, router2). I want to only index the “host” (host1 and host2) files and exclude any router files.   Additionally, there are appserver and system directories which reside under each host directory. Conceptually, you want to do the following:

* Tell Splunk to monitor the /storage/datacenter directory
* Set a whitelist for this input
* Edit the REGEX to match all files that contain “host” in the underlying path
* Edit the REGEX to match all files that contain “webserver” in the underlying path
* Edit the REGEX to match all files that start with “200906″
* Edit the REGEX to machh all files that end with “.gz”

Your final stanza in the $SPLUNK_HOME/etc/system/local/inputs.conf file would resemble the following:

[monitor:///storage/datacenter/]
sourcetype=gzfiles
_whitelist=host[^/]*/webserver/[^/]*200906[^/]*\.gz$

The above stanza would index the following files:

/storage/datacenter/host1/webserver/20090601.gz
/storage/datacenter/host1/webserver/20090602.gz
/storage/datacenter/host2/webserver/20090601.gz
/storage/datacenter/host2/webserver/20090602.gz

The above stanza would NOT index the following files or directories:

/storage/datacenter/logfile.txt
/storage/datacenter/router1/logfile.log
/storage/datacenter/host1/appserver/20090601.gz
/storage/datacenter/host2/webserver/20090601.txt

The following doc was referenced and can be viewed for more details: http://www.splunk.com/base/Documentation/latest/Admin/WhitelistAndBlacklistRules

I recently blogged about a cool open source tool which is a Splunk Dashboard. In less than an hour, you could easily bring up a central dashboard to visually oversee Splunk administration duties. Here is a basic review of how to get the dashboard working, in combination with the Check Splunk tool.

Prerequesites:

• spdash
• checksplunk
• crontab competency
• ssh competency
• web server competency
• cgi-bin competency

Even if you are not very familiar with the above items, there is plenty of information available on the web to get things going. The README files that come along with the tools are very useful and should be reviewed before proceeding. The following steps are an outline of what I performed to get the dashboard working:

Step 1: Install the spdash software on the web server host

• Installed onto my linux server splunkdemo1
• Installation consisted of: enabling the web server and placing the spdash scripts into the cgi-bin location
• Runs on top of the OS installed apache web server from /var/www/cgi-bin/spdash
• Runs on port 80
• Edited the spdash script so that $STAT directory is located in /opt/demos/splunkdash/status
• Create the above directory so that it contains ALL of the files used to compose spdash. Logs, statistics, etc… are here

Step 2: Install the checksplunk software on the Splunk server

• Installed onto my linux server splunkdemo1
• Installation consisted of: placing the checksplunk script in it’s own directory, creating a directory to store results, and enabling a local crontab to run checksplunk on a regular interval (see step 3 for the example command)
• OPTIONAL - Install checksplunk onto your other Splunk servers. My example uses hosts located at 10.1.1.1 and 10.1.1.2)

Step 3: Retrieve the checksplunk data

• Setup a crontab on the web server host to retrieve the checksplunk data

My crontab on splunkdemo1 is as follows:

splunkdemo1>crontab -l
*/5 * * * * /opt/demos/splunkdash/j2ee/checksplunk spdash
*/7 * * * * /opt/demos/splunkdash/email/checksplunk spdash
*/8 * * * * scp root@10.1.1.1:/opt/splunkdash/status/interop* /opt/demos/splunkdash/status/
*/6 * * * * scp root@10.1.1.2:/opt/splunkdash/status/cmdemo* /opt/demos/splunkdash/status/

You will notice that I am running two remote secure copies and two local checksplunk commands. The local checksplunks are configured to feed data to the /opt/demos/splunkdash/status directory.

Once you have checksplunk data feeding to the status directory, the cgi script should immediately pickup the data.

I was recently given access to an open source tool called spdash.  This tool allows you to externally visualize Splunk health from an Administrative standpoint. It consists of some cgi code and leverages a set of scripts (checksplunk) that grabs health information from one or more Splunk instances.   Information such as basic process status, listings of event counts, user specific search counts, and error messages are all presented in an intuitive screen.  Check out the main dashboard page:

Read More »