BOOK EXCERPT: When to use “transaction” and when to use “stats”

EXCERPT FROM “EXPLORING SPLUNK: SEARCH PROCESSING LANGUAGE (SPL) PRIMER AND COOKBOOK”. Kindle/iPad/PDF available for free, and hardcopy available for purchase at Amazon.

There are several ways to group events with the Search Processing Language (SPL). The most common approach uses either the transaction or stats command. But when should you use transaction and when should you use stats?

The rule of thumb: If you can use stats, use stats. It’s faster than transaction, especially in a distributed environment. With that speed, however, comes some limitations. You can only group events with stats if they have at least one common field value and if you require no other constraints. Typically, the raw event text is discarded.

Like stats, the transaction command

» Continue reading

Book Excerpt: Finding Specific Transactions

EXCERPT FROM “EXPLORING SPLUNK: SEARCH PROCESSING LANGUAGE (SPL) PRIMER AND COOKBOOK”. Kindle/iPad/PDF available for free, and hardcopy available for purchase at Amazon.

Problem

You need to find transactions with specific field values.

Solution

A general search for all transactions might look like this:

          sourcetype=email_logs | transaction userid

Suppose, however, that we want to identify just those transactions where there is an event that has the field/value pairs to=root and from=msmith. You could use this search:

   sourcetype=email_logs
   | transaction userid
   | search to=root from=msmith

The problem here is that you are retrieving all events from this sourcetype (potentially billions), building up all the transactions, and then throwing 99% of the data right in to the bit bucket. Not only is it …

» Continue reading

Removing Duplicate Consecutive Events

EXCERPT FROM “EXPLORING SPLUNK: SEARCH PROCESSING LANGUAGE (SPL) PRIMER AND COOKBOOK”. Kindle/iPad/PDF available for free, and hardcopy available for purchase at Amazon.

Problem

You want to group all events with repeated occurrences of a value in order to remove noise from reports and alerts.

Solution

Suppose you have events as follows:

          2012-07-22 11:45:23 code=239
          2012-07-22 11:45:25 code=773
          2012-07-22 11:45:26 code=-1
          2012-07-22 11:45:27 code=-1
          2012-07-22 11:45:28 code=-1
          2012-07-22 11:45:29 code=292
          2012-07-22 11:45:30 code=292
          2012-07-22 11:45:32 code=-1
          2012-07-22 11:45:33 code=444
          2012-07-22 11:45:35 code=-1
          2012-07-22 11:45:36 code=-1

Your goal is to get 7 events, one for each of the code values in a row: 239, 773, -1, 292, -1, 444, -1. You might be tempted to use the transaction command as follows:

          ... 

» Continue reading

Transaction Searching: Unifying Field Names

EXCERPT FROM “EXPLORING SPLUNK: SEARCH PROCESSING LANGUAGE (SPL) PRIMER AND COOKBOOK”. Kindle/iPad/PDF available for free, and hardcopy available for purchase at Amazon.

Problem

You need to build transactions from multiple data sources that use different field names for the same identifier.

Solution

Typically, you can join transactions with common fields like:

          ... | transaction username

But when the username identifier is called different names (login, name, user, owner, and so on) in different data sources, you need to normalize the field names.

If sourcetype A only contains field_A and sourcetype B only contains field_B, create a new field called field_Z which is either field_A or field_B, depending on which is present in an event. You can then build the transaction

» Continue reading

Splunk Book Excerpt: Finding Metrics That Fell by 10% in an Hour

EXCERPT FROM “EXPLORING SPLUNK: SEARCH PROCESSING LANGUAGE (SPL) PRIMER AND COOKBOOK”. Kindle/iPad/PDF available for free, and hardcopy available for purchase at Amazon.

Problem

You want to know about metrics that have dropped by 10% in the last hour. This could mean fewer customers, fewer web page views, fewer data packets, and the like.

page91image14920
page91image15192
page91image15464
page91image15736
page91image16008

Solution

To see a drop over the past hour, we’ll need to look at results for at least the past two hours. We’ll look at two hours of events, calculate a separate metric for each hour, and then determine how much the metric has changed between those two hours. The metric we’re looking at is the count of the number of events between two hours ago and

» Continue reading

Splunk Book Excerpt: Grouping Events

EXCERPT FROM “EXPLORING SPLUNK: SEARCH PROCESSING LANGUAGE (SPL) PRIMER AND COOKBOOK”. Kindle/iPad/PDF available for free, and hardcopy available for purchase at Amazon.

Grouping Events

There are several ways to group events. The most common approach uses either the transaction or stats command. But when should you use transaction and when should you use stats?

The rule of thumb: If you can use stats, use stats. It’s faster than transaction, especially in a distributed environment. With that speed, however, comes some limitations. You can only group events with stats if they have at least one common field value and if you require no other constraints. Typically, the raw event text is discarded.

Like stats, the transaction command can group events based

» Continue reading

New Splunk Book Excerpt: Finding Events After Other Events

EXCERPT FROM “EXPLORING SPLUNK: SEARCH PROCESSING LANGUAGE (SPL) PRIMER AND COOKBOOK”. Kindle/iPad/PDF available for free, and hardcopy available for purchase at Amazon.

Finding Events After Other Events

Problem

You need to get the first 3 events after a particular event (for example, a login event) — for the events related to a particular user — but there is no well-defined ending event.

Solution

Given the following ideal transaction that starts with a login action:

          [1] 10:11:12 userid=root action=login
          [2] 10:11:13 userid=root action=”cd /”
          [3] 10:11:14 userid=root action=”rm -rf *”
          [4] 10:11:15 userid=root server=”echo lol”

The obvious search choice is to use transaction that starts with the login action:

          ... | transaction userid startswith="(action=login)" maxevents=4

The problem is that you will

» Continue reading