Universal or Heavy, that is the question?


As a Professional Services Consultant, a discussion that I often encounter when on site with customers is whether to use a Universal Forwarder or a Heavy Forwarder.

Splunk provides two different binaries, the full version of Splunk and the Universal Forwarder. A full Splunk instance can be configured as a Heavy Forwarder.  The Universal Forwarder is a cut down version of Splunk, with limited features and a much smaller footprint.

I am going to show in this blog why Splunk Professional Services recommend the use of Universal Forwarders in preference to Heavy Forwarders whenever possible to ensure a faster, more efficient Splunk Platform.

When should the Universal Forwarder be used and why?

The Universal Forwarder is ideal for collecting files from disk (e.g. a syslog …

» Continue reading

Splunk your Google Analytics

Gain more insight into site performance and user activity by correlating Google Analytics data within Splunk.

A customer of mine recently wanted to understand more about the journey that retail consumers take when they arrive at its website. They recognized that consumers who have previously bought from the site will have more familiarity with the design and layout than those visiting the site for the first time. In addition, consumers who went directly to the site would have a greater brand engagement than those who were referred from an affiliate site.

If only we could implement a method to back up the data that gets submitted to  Google Analytics, also sending it back to the local Apache web server logs …

» Continue reading

What size should my Splunk license be?

This is a pretty common question in Splunkland. Maybe you’re an admin wondering how much license you’ll need to handle this new data source you have in mind for a great new use case. Or you’re a Splunker trying to answer this question for a customer. Or a partner doing the same. Given how often this comes up, I thought I’d put together an overview of all the ways you can approximate how big a license you need, based on a set of data sources. This post brings together the accumulated wisdom of many of my fellow sales engineers, so before I begin, I’d like to thank them all for the insights and code they shared so willingly. Thank you

» Continue reading

Your Splunk Sandbox

When I was an admin, sometimes I wanted to Splunk things, but not in my production environment. Maybe I wanted to add data and define the corresponding sourcetype. Maybe I wanted to mess with some backend conf files. Maybe I wanted to muck around with a new version of a search or dashboard. Whatever the reason, I learned a few approaches that may be obvious for the Splunk Ninjas out there, but not so much for our adorable n00bs. Either way, if you find yourself hesitating to try something Splunky, then this post is for you.

Build a Splunk Sandbox

Ideally, you’re installing Splunk on your local workstation (desktop/laptop), but if your company hasn’t given you access rights to install Splunk, then see if …

» Continue reading

Writing Actionable Alerts

Is your Splunk environment spamming you? Do you have so many alerts that you no longer see through the noise? Do you fear that your Splunk is losing its purpose and value because users have no choice but to ignore it?

I’ve been there. I inherited a system like that. And what follows is an evolution of how I matured those alerts from spams to saviors.

Let it be known that Splunk does contain a number of awesome search commands to help with anomaly detection. If you enjoy what you read here, be sure to check them out since they may simplify similar efforts. http://docs.splunk.com/Documentation/Splunk/latest/SearchReference/Commandsbycategory#Find_anomalies

Stage 1: Messages of Concern

Some of the first alerts created are going to be searches …

» Continue reading

How’s my driving?

It was the summer of 2014. I was well into my big data addiction thanks to Splunk. I was looking for a fix anywhere: Splunk my home? Splunk my computer usage? Splunk my health? There were so many data points out there for me to Splunk but none of them would payoff like Splunking my driving…

Rocky Road

At the time, my commute was rough. Roads with drastically changing speeds, backups at hills and merges, and ultimately way more stop and go than I could stomach. But how bad was my commute? Was I having as bad an impact on the environment as I feared? Was my fuel efficiency much worse than my quiet cruise-controlled trips between New York and Boston? …

» Continue reading

Fixing Scripted Inputs in Tiered Deployments

The Splunk App for Microsoft Exchange has a useful lookup named ad_username. It takes the various forms that you can logon to a domain as (like DOMAIN\user and user@domain.com) and normalizes them. Further, it then takes all the user aliases and normalizes them so adrian.hall is the same as ahall and that is the same as adrian. It’s really useful when you are trying to deal with domain accounts from a support functionality – you don’t have to know how they logged in – only what their official username is.

AD_Username is a scripted input written in Python and lives in the bin directory of the application directory. It relies on two files that live in the local directory called …

» Continue reading

Search Command> diff

What’s the grooviest Splunk search command goin’ round? It’s diff man, can you dig it?

That’s right, diff. What other command is based on a *nix file comparison utility that’s been around since the early 70’s?

Splunk’s diff operates just like good ol’ diff does on a *nix platform – it compares two inputs and tells you what the differences are, in a very distinct format. But where *nix diff normally compares two files, Splunk’s diff compares the content of two events.

We can use diff to compare one field in an event to that same field in another event, or we can go for broke and have diff compare “_raw” – or the content of the entire event …

» Continue reading

SplunkTalk – #69 – The Walking Dead

Ok… we’re officially never again going to say “we’re back”. Except for right now. We’re back. At Splunk’s 2013 User Conference, (a.k.a. “.conf”–get it… dot conf.. our configuration files 😛 ) a number of listeners came up to us and said “Yo… when’s the podcast coming back?!?!?!?” To that we replied, “well, how about now”. So with out further adieu, I, Michael Wilde, your faithful Splunk Ninja would like to introduce an amazing new co-host of SplunkTalk, Hal Rottenberg. (That’s long o in Rottenberg, as in O my gosh he’s great). This episode of SplunkTalk returns with an overview of our favorite features in the newly released “Splunk 6.0”, and a question about a Splunk 6.0 search head …

» Continue reading

Clustering Optimizations in Splunk 6

One of the new features we introduced in Splunk 6 is the Simplified Clustering Management. This allows administrator to setup and monitor the health of the cluster through an easy to use, intuitive UI. In addition to the cool new UI, many performance optimizations were added to handle peer failures and recovery from such failures blazingly fast. In this blog post, I’m going to highlight two such performance optimizations.

1. First Searchable Copy Optimization

This optimization is all about making sure that at least one, complete searchable copy exists in the cluster so that business users can continue to use the data while the cluster master is handling peer failures.

Let’s take a look at this with an example.  Assume …

» Continue reading