Episodes are recorded live every Friday at 11AM Central Time – Email us at splunktalk@splunk.com to ask questions and have them answered on air!

With Splunk, because of Universal indexing of all the search terms in your data and search time field extraction capabilities, event correlation becomes a natural feature for it. There are multiple ways to achieve different types of event correlation within Splunk. What I will do is provide a non-exhaustive list of some of the methods that can be used to accomplish this.

Manual Event Correlation

Every time a Splunk user performs an ad-hoc search and pivots on results to find what else happened in the same time line, he or she is manually performing event correlation with time being the universal pattern to relate events. For instance, the user can use the Splunk time picker to narrow down a time and then type something as general as “error” into the search bar to search.

After receiving results, the user can then use the histogram to zoom in on a particular event’s time line and then use * as a search term to see what else happened at that particular frame in time.  Events are correlated by search using time as the pivot. This is what I call manual event correlation, which is just as important as automatic event correlation, for troubleshooting. In what follows, I will discuss the various ways Splunk can be used to automate different types of event correlation.

Transaction Search

Splunk has created “Transaction Search.” What this means is that if events have similar values for extracted fields or starting/ending terms, Splunk can automatically correlate these events as a result of a search and group the returned results. Rather than repeat what has already been said about transaction search, I encourage you to read this blog entry by Maverick for an in-depth example. You can also see my SOA article for a real world use case on using transaction search to correlate event activity across application tiers.

On the other hand, as to not to have you leave this post, I’ll provide a small example on using transaction search. On Splunkbase, you can download an app that indexes on a daily basis the world’s most malicious IP source addresses, according to one source. Here’s a gratuitous screenshot.

Needless to say, you would want to know if any of these IP addresses appeared as source IP’s in your own logs. An example search such the as one below would group events that included the offending IP addresses in your own events.

sourcetype="ip_watchlist" OR (sourcetype="sshd" login accepted)|transaction offending_ip,src_ip maxspan=1d connected=f|eval count_sourcetypes=mvcount(sourcetype)|where count_sourcetypes>1

What this search does is say if someone has logged in using SSH and their source IP is one that is in the list of malicious IP addresses (the transaction command does the grouping) within a day’s span, and the number of sourcetypes in the grouping is greater than one so we know both sourcetypes were in in the grouping, then return results. You can save this type of search, schedule it to run on an interval, and then have Splunk automatically create an alert to notify you, if events are matched.

A variant of transaction search is statistical aggregation, where numerical aggregations of different fields are grouped by other fields. Here’s a simple example usage using mail logs that counts the number of bytes coming into each relay.

sourcetype=email|stats count(bytes) as byte_count by relay|sort - byte_count

There are times where you would rather use the Splunk stats command over the transaction command and this is described here.

Sub Searches

Another way to automate event correlation is to use the concept of a sub search. If you like the approach of an outer join in database terms, note that Splunk can perform sub searches to narrow down the criteria for one event and then perform another search on the first set of results. Again, as to not repeat what has already been written, here’s an article describing this feature for Event Correlation.

Although a related feature is not used as much, If you would like to join events stored with Splunk itself, Splunk does have a join search command. There may be use cases where performing the logical union of related events is necessary.

Lookup

Splunk has the capability to correlate with data that is external to Splunk using the lookup command. The most basic use for this is when you have some fields that are in your Splunk event that need to correlate to fields in an external CSV file. At search time, Splunk will perform the look up and introduce new fields from the external CSV file as patterns are matched. In essence, this enriches your existing indexed data with external sources at search time.

If you would like to correlate events with the same field value between external database tables and events within Splunk, Splunk’s lookup command can also be used to accomplish this. The basic idea is that a user written program will be called to perform the SQL to bring in new fields at search time. This is called a dynamic lookup as opposed to the static lookup accomplished by using a CSV file. In fact, because the program is developed by the user to perform the external lookup, you are not limited to CSV files or databases to perform lookups. Anything can be correlated using the lookup command if you have programmatic access to it. Examples could include a call to a web service, an external DNS lookup, or calling whois for an IP address.

Conclusion

In this entry, I have provided a non-exhaustive list of the most common ways event correlations are performed in Splunk. There are more subtle ways to accomplish this, but since this is a blog entry, I decided to start with the most prevalent usages. The main point I want to leave you with is that event correlation and subsequent alerting can be performed on any type of events that you choose to index giving you a powerful technique to analyze, aggrandize, and interpret data.

Here are some of the greatest hits:

• PCI App (Creative Commons Version) – Peter Bassill (aka BinaryArp) hacked on a newer and much improved version 1.0 of his PCI (Payment Card Industry) app in time for our Users’ Conference that took place earlier this month. And by the end of this month, he had already added 2 more revisions culminating in version 1.2 uploaded on 8/29. If you need to satisfy PCI compliance requirements, you might want to take a look.
• Splunk for Nagios - Luke Harris posted his Splunk for Nagios add-on just as I was thinking about writing one myself. This is something quite a few users have requested, and now we have it, thanks to Luke. With this app, you can search Nagios alerts and notifications and graph problems over time.
• Google Maps for Splunk – This add-on provides a module to display events/results of Splunk searches on a Google map. Thanks to Siegfried Puchbauer for writing up this one.

One of these may win the grand prize… or not! The contest hasn’t ended yet, and we haven’t commenced voting.

Think you have what it takes? Submit your app or add-on to Splunkbase and be eligible to win.

Episodes are recorded live every Friday at 11AM Central Time – Email us at splunktalk@splunk.com to ask questions and have them answered on air!

Enjoy listening!

Episodes are recorded live every Friday at 11AM Central Time – Email us at splunktalk@splunk.com to ask questions and have them answered on air!

Enjoy Listening

The Splunk Answers juggernaut is racing past milestones. We launched Answers in early April to little fanfare. In June we reached the 1,000 question milestone. And in late July we reached the 2,000 question milestone. And on August 18, 2010, Splunk Answers surpassed 1,000 users. We had no idea when we started Answers that it would catch on this rapidly. Just two weeks ago we reached 800 users, and usage has spiked since then. I wish I could say we had to make a special effort for Splunk Answers to reach the heights it has, but the truth is that it has caught on organically with our users who found it very useful to get quick answers to their questions. Fancy that.

Splunk Answers Screenshot

Top Community Users

While we’re at it, I’d like to take a moment and single out our top community contributors on Splunk Answers.

Special mention goes to Lowell (from Martin’s Famous Pastry Shoppe), ftk (from CallMiner), and muebel (from Netsmart), who all attended our glorious users’ conference – it was great meeting you in person.

Yesterday, (August 18, 2010) I was on the phone with some folks evaluating Splunk and they asked me a ton of questions. Many of them were about architecture, but some were about search. They asked “how to I make Splunk look a bit more like the Windows Event Viewer?”, “how do I do alerts in a smarter way?”, and “are there ways to make search faster and more efficient?” Verbally, I gave them some hints and said “I’ll send you some screenshots”. Well, you know Wilde (if you don’t, he’s the Splunk Ninja guy who likes to make videos)–screenshots just won’t do for this one.

I sat down with my handy copy of Telestream’s Screenflow and my new Heil PR-40 microphone and shot this video for everyone who might find it interesting.

Tips and Tricks: Search, UI, Filtering and Alerting.
Its a good video for beginners that will show you how to do basic search, but quickly gets in to how fields, event display, filtering, slightly advanced search and alerting works. Even if you have used Splunk for a while, there might be a few things you will learn from this. If you are new to Splunk, do watch this–it will shorten your learning curve.

Fullscreen the video. It will look much better as Splunk has a large web UI.

For those who might have a flash blocker installed preventing the player from loading in your browser (as the movie is right under this message), here is a link to the movie.

Some of the noteworthy sessions I attended were the session by Corporate Express Australia ( Staples in Australian:) and the sessions done by two of the hottest companies in Silicon Valley, VMware and Salesforce. San Francisco might have been in its usual summer denial, but it was warm and sunny in the Palace hotel, as we learned a great deal from these really capable folks.

First, a word about  Shaun Butler and Luke Harris of Corporate Express. What I found out really quickly about them was the fact that they must’ve splunked pretty much everything they got their hands on. EMC Symmetrix logs, SAP logs, webmethods logs, netflow data, VMware ESX logs and VC events/audit trails, Nagios data, Windows event logs, Active Directory, Altiris logs, DNS, DHCP, Postfix,  web logs….the list goes on.  By the end of the first day, if anyone asked me any question like “Has anyone Splunked “x” – my default answer started with “Check with the CE guys, they are sure to have done it:)”. The most impressive thing about their session, was not however the breadth of technology. It was what they had accomplished with the data.  Here’s an example:

They had figured out at a very granular level which component in their infrastructure was providing what type of response times to a user, using data across their application, network and infrastructure.

They had also tied together and analyzed data from various different sources to perform a solid cost analysis and capacity planning for their SAP environment with reports like Average WAN bandwith usage per application and even average WAN bandwidth footprint per user! Really really impressive. More about CE in Sanjay’s blog here

The VMware session was also remarkable. In Alan Burnett’s words, this year VMware will be at almost $3Bn in revenue – the order processing system is mission critical, to say the least. With Splunk, they have been able to provide much needed visibility into various disparate systems that play in order processing and correlate information across all of them. They not only Splunk their Oracle RAC, Fusion Middleware  and various Weblogic based customer facing applications, they are also in process of trying out our soon-to-be –in-beta VMware app for vSphere and have a long to do list for the year comprising: email, SOX compliance, Spring TC Server, VMware View etc etc. We look forward to it!

The session by Salesforce.com was quite noteworthy too. If there is a quantum change going on in the CRM space, Salesforce is leading it. And they are being ably supported by Splunk. Narayan Bharadwaj from Salesforce showed this very interesting data showcasing their explosive Splunk usage (we’re rather proud of it too!:). In the space of 7 months, Salesforce has increased the number of searches it runs with Splunk by an order of magnitude (from 4370 to almost 77,000!) and has almost 300 people in the organization using Splunk to answer questions like: What are the performance hotspots in a customer’s application?  What is the usage of Chatter looking like for any particular customer? Which users are playing with sensitive data?

Of course this was just a small snapshot. There were many many other sessions that had gripping content, great presenters and phenomenal ideas. When .conf 2011 rolls around, be sure to be there and catch the action in person!

Enjoy Listening

Overview

1. Setup a Notifo account.
2. Install the Notifo app on your iPhone.
3. Install the notifo.py Python module.
4. Install the splunknotifo.py Python alert script.
5. Setup splunknotifo.py
6. Setup saved search.

Assumptions

• This process assumes that you’ve got Splunk installed and monitoring a file containing sshd log messages.

Steps

1. Browse to https://notifo.com/user/register to setup a Notifo account.
2. Browse to https://notifo.com/user/login, login, and visit the Settings page.
3. Locate and record your API Secret (screenshot)
4. From your desktop or your iPhone browse to the iTunes App Store to install and configure the Notifo app.
5. On the system running Splunk, download and install the notifo.py Python module (screenshot):
~$ cd /usr/local/src
/usr/local/src$ git clone git://github.com/mrtazz/notifo.py.git
/usr/local/src$ cd notifo.py
/usr/local/src/notifo.py$ $SPLUNK_HOME/bin/splunk cmd python setup.py install

6. On the system running Splunk, download the splunknotifo.py Python alert script (screenshot):
~$ cd $SPLUNK_HOME/bin/scripts
/opt/splunk/bin/scripts$ get http://github.com/ampledata/soss/raw/master/splunknotifo/splunknotifo.py
/opt/splunk/bin/scripts$ get http://github.com/ampledata/soss/raw/master/splunknotifo/splunknotifo_conf-default-.py

7. Configure splunknotifo_conf.py with your Notifo APIUsername and APISecret (see step #3 above):
~$ cd $SPLUNK_HOME/bin/scripts
/opt/splunk/bin/scripts$ mv splunknotifo_conf-default-.py splunknotifo_conf.py
/opt/splunk/bin/scripts$ vim splunknotifo_conf.py

8. Using the Splunk web interface, search for the term(s) you’d like to match and click Actions >> Save search… (screenshot).
9. Enter the parameters for your Saved Search:
   
10. Done!

To Test

1. Generate some sshd log messages.
2. You should get an alert on your iPhone like this: