Distributed searching

If you are a long time enterprise user of the 3.x product, you may have become used to the pull-down menu for distributed searching. One of the common use cases for this menu was searching specific indexers in your distributed search. A common question was: “Can we restrict the server via search syntax?”. In the 3.3 and 3.4 product, you cannot restrict via syntax through the web interface. There is a trick you can use via the command line, but that doesn’t help when you want to do this in a saved search.

In the 4.0 release, we have removed the pull-down menu and implemented indexer restrictions with search syntax. The new parameter is called “splunk_server”. Let’s assume I have a…

» Continue reading

Monitoring input files with a white list

There are many ways to feed data into Splunk. One method is to monitor the files within a directory. In the default ‘monitor’ configuration, Splunk will try to index all files within a specified directory. In some cases, you may have a directory which contains many files including some that you do not want to index. Splunk can be configured to index specific file types as well as sub directories. Here is a real-world working example of how to use a white list…

Let us assume we want to index certain compressed files (*.gz) where the file name starts with “200906″. One of the filename’s is “20090631.gz”. These files exist in a specific directory: “/storage/datacenter/host1/webserver”. To make things more interesting, I…

» Continue reading

Splunk Dashboards outside of Splunk (part 2)

I recently blogged about a cool open source tool which is a Splunk Dashboard. In less than an hour, you could easily bring up a central dashboard to visually oversee Splunk administration duties. Here is a basic review of how to get the dashboard working, in combination with the Check Splunk tool.

Prerequesites:

  • spdash
  • checksplunk
  • crontab competency
  • ssh competency
  • web server competency
  • cgi-bin competency

Even if you are not very familiar with the above items, there is plenty of information available on the web to get things going. The README files that come along with the tools are very useful and should be reviewed before proceeding. The following steps are an outline of what I performed to get the dashboard working:

Step 1: Install the spdash software on the…

» Continue reading

Splunk Dashboards outside of Splunk

I was recently given access to an open source tool called spdash. This tool allows you to externally visualize Splunk health from an Administrative standpoint. It consists of some cgi code and leverages a set of scripts (checksplunk) that grabs health information from one or more Splunk instances. Information such as basic process status, listings of event counts, user specific search counts, and error messages are all presented in an intuitive screen. Check out the main dashboard page:

» Continue reading