Reflections on a Splunk developer’s journey : Part 1

It seems like only yesterday

…that I was writing my first Splunk App. It was the openness and extensibility of the Splunk platform that attracted me to this pursuit in the first place, and when I discovered the thriving community on Splunkbase (now called Splunk Apps / Answers), I just had to contribute. 11,000+ downloads across 9 different freely available community offerings later, I am feeling somewhat reflective. So in this 2 part blog series I want to share with you some of my lessons learned from developing and supporting Splunk community Apps/Add-ons (part 1) and then some musings on why you should consider developing Splunk Apps/Add-ons yourself and contribute to the Splunk developer ecosystem (part 2).

Some lessons learned…

» Continue reading

Splunk App for VMware v3.1: Transforming operational visibility into virtualized datacenters with built-in storage correlation

Earlier today, we announced the general availability of the latest release version 3.1 of the Splunk App for VMware. This release is monumental providing radical cross-tier insights into your virtual infrastructure. In this latest release, we’ve focused on improving time-to-value with 3 important features – correlated insights, adaptable reporting and enhanced topology map. Let’s delve a little deeper into each area.

1. Built-in correlation between VMware and storage environments:

If you’re running a virtual datacenter, storage latency in virtual environments is one of the most common performance issues you are likely to deal with. Storage I/O latencies impact performance of VMs because read/write operations can cause performance issues to the shared resources in your datacenter; as VMs contend with each …

» Continue reading

Detecting Windows XP Systems with Splunk

Windows XP is dead! Soon after Windows XP was introduced, Microsoft introduced the Trustworthy Computing Initiative – a kind of “security first” thinking that has been the hallmark of Microsoft for the last decade. Prior to the security focus, Microsoft operating systems were well known as a leaky sieve for viruses. Now, 12 years later, Windows XP is finally ready to be dropped. Well, to be honest – that happened a few years back. But many people are holding on to their XP installs for one reason or another. Now it’s time to give them up.

How can you tell who is connecting to your facilities with Windows XP systems? There are a variety of ways depending on if they …

» Continue reading

Splunk as a Recipient on the JMS Grid

A number of years ago, I was fascinated by the idea of SETI@home. The idea was that home computers, while idling, would be sent calculations to perform in the search for extraterrestrial life. If you wanted to participate, you would register your computer with the project and your unused cycles would be utilized for calculations sent back to the main servers. You could call it a poor man’s grid, but I thought it of it as a massive extension for overworked servers. I thought the whole idea could be applied to the Java Messaging Service (JMS) used in J2EE application servers.


Almost a decade ago, I would walk around corporations at “closing” time and see a mass array …

» Continue reading

Another NY Metro Splunk Users Group Meeting

We had our first NY Metro Splunk Users Group meeting of the year this week and it was hosted at Blackrock in NYC with Reed Kelly, one of the leaders of the users group playing host. Thanks Reed.

Our first order of business was to watch a presentation from Splunk Product Manager Jack Coates on the new 3.0 Splunk Common Information Model. Unlike the past CIM that focused heavily on security, the new CIM is general purpose for all of IT and flexible to add more knowledge to it, when needed. As a bonus, the app in the app store has data models to quickly get started and test your data sources.

Next, we had a discussion (or some …

» Continue reading

Running two Universal Forwarders on Windows

We get quite a few requests on how to run two Splunk Universal Forwarders on the same Windows host. Why would you do this? The primary reason is that you have a lab environment and want to compare one version of Splunk to another during an evaluation of a new version. You may also have two sets of files you need to ingest into Splunk and the files have differing access permissions such that Splunk needs to run as different users. It’s really an edge case and definitely not something you want to generally do in production.

In Linux, this is a fairly simple process – just install to a different directory and change the ports and you are done. …

» Continue reading

Search Command> stats, eventstats and streamstats

Getting started with stats, eventstats and streamstats

When I first joined Splunk, like many newbies I needed direction on where to start. Someone gave me some excellent advice:

“Learn the stats and eval commands.”

Putting eval aside for another blog post, let’s examine the stats command. It never ceases to amaze me how many Splunkers are stuck in the “super grep” stage. They just use Splunk to search (happily I might add) for keywords and phrases over many sources of machine data. Hopefully this will help advance some folks beyond “super grep” as well as assist those who may be new to Splunk.

When you dive into Splunk’s excellent documentation, you will find that the stats command has a couple …

» Continue reading

What’s new in Microsoft Apps

Splunk is exhibiting at the Microsoft Exchange Conference this week. If you are in town, please stop by booth #805 in the Eastside to see us. To coincide with this conference, we are releasing a whole slew of new apps and add-ons. Here are some of the highlights:

The Splunk App for Microsoft Exchange has undergone a huge makeover and now includes complementary functionality from the Active Directory Domain Services and Windows realm. We can correlate across those three platforms to see new and unique things. Want to understand how a Windows update affected the performance of your Exchange hosts? Now you have the information available to you. Want to arrange the app panels in ways that are useful to …

» Continue reading

Time based load balancing – Part 2

This is a follow up to my earlier post on the forceTimebasedAutoLB setting for outputs.conf.

There was some discussion (read: prove it to me) on the IRC channel about how would this feature behave with multi-line events or double byte characters. Well, you will be glad to know it worked flawlessly.

My events are from a Japanese Windows instance:

Screen Shot 2014-03-24 at 5.48.31 PM

I sent over 500,000 events using the oneshot command from the UF.

Screen Shot 2014-03-24 at 5.49.33 PM

And it worked as expected.

Lastly, there was some talk about data munging. Meaning part of one event being incorrectly added to another event. This can happen when Splunk doesn’t break a multi-line event proper. In my test, I didn’t even setup a BREAK_ONLY_BEFORE or LINE_BREAKER rule on the …

» Continue reading

Using Splunk as a data store for developers

A number of years ago, I wrote a blog entry called Everybody Splunk with the Splunk SDK, which succinctly encouraged developers to put data into Splunk for their applications and then search on the indexed data to avoid doing sequential search on unstructured text. Since it’s been a while and I don’t expect people to memorize the dissertations of ancient history (to paraphrase Bob Dylan), I’ve decided to write about the topic again, but this time in more detail with explanations on how to proceed.

Why Splunk as a Data Store?

Some may proclaim that there are many no-sql like data stores out there already, so why use Splunk for an application data store? The answers point to simplicity, …

» Continue reading