Simeon's Blog

40 Days of 4.0: Distributed searching

simeon — Thu, 30 Jul 2009 20:09:06 +0000

If you are a long time enterprise user of the 3.x product, you may have become used to the pull-down menu for distributed searching. One of the common use cases for this menu was searching specific indexers in your distributed search. A common question was: “Can we restrict the server via search syntax?”. In the 3.3 and 3.4 product, you cannot restrict via syntax through the web interface. There is a trick you can use via the command line, but that doesn’t help when you want to do this in a saved search.

In the 4.0 release, we have removed the pull-down menu and implemented indexer restrictions with search syntax. The new parameter is called “splunk_server”. Let’s assume I have a distributed searcher (hostname=searcher1) and three indexers (hostname=indexer1, hostname=indexer2, and hostname=indexer3). If I am searching for “error” and my goal is to restrict my searches to indexer3, I would use the following query:

splunk_server=indexer3 error

To search anything but indexer3 I would use:

error NOT splunk_server=indexer3

Using this restriction can be useful for tracking specific datacenters, monitoring server health, and securing data (can add this as a filter to a role). For the complete documentation on this command, see our official documentation:

http://www.splunk.com/base/Documentation/latest/User/SpecifyMultipleServersToSearch

Note: distributed searching is limited to the Splunk enterprise version.

Monitoring input files with a white list

simeon — Thu, 09 Jul 2009 21:46:31 +0000

There are many ways to feed data into Splunk. One method is to monitor the files within a directory. In the default ‘monitor’ configuration, Splunk will try to index all files within a specified directory. In some cases, you may have a directory which contains many files including some that you do not want to index. Splunk can be configured to index specific file types as well as sub directories. Here is a real-world working example of how to use a white list…

Let us assume we want to index certain compressed files (*.gz) where the file name starts with “200906″. One of the filename’s is “20090631.gz”. These files exist in a specific directory: “/storage/datacenter/host1/webserver”. To make things more interesting, I have other *.log files in that directory. There are also other subdirectories within datacenter (such as host2, router1, router2). I want to only index the “host” (host1 and host2) files and exclude any router files. Additionally, there are appserver and system directories which reside under each host directory. Conceptually, you want to do the following:

* Tell Splunk to monitor the /storage/datacenter directory
* Set a whitelist for this input
* Edit the REGEX to match all files that contain “host” in the underlying path
* Edit the REGEX to match all files that contain “webserver” in the underlying path
* Edit the REGEX to match all files that start with “200906″
* Edit the REGEX to machh all files that end with “.gz”

Your final stanza in the $SPLUNK_HOME/etc/system/local/inputs.conf file would resemble the following:

[monitor:///storage/datacenter/]
sourcetype=gzfiles
_whitelist=host[^/]*/webserver/[^/]*200906[^/]*\.gz$

The above stanza would index the following files:

/storage/datacenter/host1/webserver/20090601.gz
/storage/datacenter/host1/webserver/20090602.gz
/storage/datacenter/host2/webserver/20090601.gz
/storage/datacenter/host2/webserver/20090602.gz

The above stanza would NOT index the following files or directories:

/storage/datacenter/logfile.txt
/storage/datacenter/router1/logfile.log
/storage/datacenter/host1/appserver/20090601.gz
/storage/datacenter/host2/webserver/20090601.txt

The following doc was referenced and can be viewed for more details: http://www.splunk.com/base/Documentation/latest/Admin/WhitelistAndBlacklistRules

Splunk Dashboards outside of Splunk (part 2)

simeon — Mon, 22 Jun 2009 22:29:52 +0000

I recently blogged about a cool open source tool which is a Splunk Dashboard. In less than an hour, you could easily bring up a central dashboard to visually oversee Splunk administration duties. Here is a basic review of how to get the dashboard working, in combination with the Check Splunk tool.

Prerequesites:

spdash
checksplunk
crontab competency
ssh competency
web server competency
cgi-bin competency

Even if you are not very familiar with the above items, there is plenty of information available on the web to get things going. The README files that come along with the tools are very useful and should be reviewed before proceeding. The following steps are an outline of what I performed to get the dashboard working:

Step 1: Install the spdash software on the web server host

Installed onto my linux server splunkdemo1
Installation consisted of: enabling the web server and placing the spdash scripts into the cgi-bin location
Runs on top of the OS installed apache web server from /var/www/cgi-bin/spdash
Runs on port 80
Edited the spdash script so that $STAT directory is located in /opt/demos/splunkdash/status
Create the above directory so that it contains ALL of the files used to compose spdash. Logs, statistics, etc… are here

Step 2: Install the checksplunk software on the Splunk server

Installed onto my linux server splunkdemo1
Installation consisted of: placing the checksplunk script in it’s own directory, creating a directory to store results, and enabling a local crontab to run checksplunk on a regular interval (see step 3 for the example command)
OPTIONAL - Install checksplunk onto your other Splunk servers. My example uses hosts located at 10.1.1.1 and 10.1.1.2)

Step 3: Retrieve the checksplunk data

Setup a crontab on the web server host to retrieve the checksplunk data

My crontab on splunkdemo1 is as follows:

splunkdemo1>crontab -l
*/5 * * * * /opt/demos/splunkdash/j2ee/checksplunk spdash
*/7 * * * * /opt/demos/splunkdash/email/checksplunk spdash
*/8 * * * * scp root@10.1.1.1:/opt/splunkdash/status/interop* /opt/demos/splunkdash/status/
*/6 * * * * scp root@10.1.1.2:/opt/splunkdash/status/cmdemo* /opt/demos/splunkdash/status/

You will notice that I am running two remote secure copies and two local checksplunk commands. The local checksplunks are configured to feed data to the /opt/demos/splunkdash/status directory.

Once you have checksplunk data feeding to the status directory, the cgi script should immediately pickup the data.

Splunk Dashboards outside of Splunk

simeon — Thu, 21 May 2009 22:32:16 +0000

I was recently given access to an open source tool called spdash. This tool allows you to externally visualize Splunk health from an Administrative standpoint. It consists of some cgi code and leverages a set of scripts (checksplunk) that grabs health information from one or more Splunk instances. Information such as basic process status, listings of event counts, user specific search counts, and error messages are all presented in an intuitive screen. Check out the main dashboard page:

After installing and running it internally on some of our systems, I have come away very impressed with what this can do for the System Administrator of a Splunk instance. One of the great features is the server link which allows you to get specific server information. Here is a screen capture of that screen:

When I first saw this being developed, I thought that it might be challenging to deploy. After less than an hour, I had a handful of servers sending and updating data to this dashboard. Now it’s no cakewalk, but it’s pretty straighforward. If you are very familiar with Splunk, have scripting experience, and can manage cgi on a web server then you should have no trouble. Kudos to the author, Kirk Waingrow, for making this available to the general public! If you are a System Administrator and manage Splunk, I would highly recommend you check this out.

I will post a follow up that will contain details on my deployment…