Blogs

Splunk is on track to become a billion-dollar company and you, the intrepid sysadmin/developer, are going to help us get there. Now, this is not a statement that I’m making as an analyst who “covers” the enterprise software market, and compiles a list of “top software companies to watch”. I’m writing this as Splunk’s Platform Architect, a techie whose goals are to ensure that what comes out of our development group is compelling and exciting to those that are actually working with the product.

It is this developer-centric ethos that sets us apart from so many of the other enterprise software firms and has already paid dividends on community goodwill. Instead of making prospective buyers jump through registration hoops just to…

bash.org is a natural dataset for splunking. It’s a huge blob of loosely structured text data, and it’s made of win.

To play with a live instance, go to bash.splunklabs.com, login: guest, password: guest.

Of course, Splunk duplicates the functionality of the site itself. We can find, for example, the top 100 IRC quotes:

Splunk lets us do considerably more, though. What are the top one-liners?

How many more quotes mention “girlfriend” than “boyfriend”, i.e. exactly how bad is this sausage party?

Are there any commonly quoted individuals?

Are there any interesting trends in quote scores over time? Take a look at high quote scores vs. quote ID:

It seems likely that older quotes, especially good ones, benefit from a disproportionately greater number of views (the rich…

The following is copped from a support email by Stephen Sorkin who is the man behind the splunk server curtain … thought it should go broader.

I’m the manager of the search and indexing team at Splunk. We’re still in the process of writing up our findings from storage benchmarks but here are the general details.

High IO/s typically means both faster indexing in general and faster searching of rare, temporally incoherent events. On average, we’ve seen indexing speeds increase by about 66% going from an 7200 RPM SATA RAID to a 15K RPM SCSI RAID. We’ve seen comparable performance from SCSI and SAS RAIDs, provided they’re 15K RPM.

The best best benchmarking tool we’ve found for measuring how Splunk will behave on…

Hey all,

It’s taken longer than we would have liked but our 3rd preview build has been posted.
Get’um here

A bunch of work has gone into windows stability, tons of bugs were fixed, and a bunch of customer requests have been implemented ( we will let you know out of band ). We expect that this release should be more stable, slightly faster, and less buggy.

Left to do, we still have a bunch of IE work, performance improvements, and cleaning up of some features like interactive field extraction and event type discovery.

Its still not production ready so don’t even think of trying it out for real – and there is no guarantee that migration will work from a preview to GA (…

For the past year I have been working on a book about visualization. It will be called “Applied Security Visualization“. The book is going to talk about all the aspects of visualizing security data. Anything from important data sources and graphs to use-cases and open source tools for visualization. The main use-cases I write about evolve around Perimeter Threat, Compliance, and Insider Threat.

Last year during RSA, Addison-Wesley (my publisher) recorded some videos, where I talk about the book and some of its contents. Here are the links to the videocasts:

• Security visualization
• Bridging security and visualization

At this point, I have one more chapter to write before the book is done. A rough-cut version should be available by RSA this year and the…

I have a Google alert set up to email me news of the extraordinary concerning Splunk. Most of them are press releases by either us or our agency, which are all well and fine (this is how most companies seed stories anyway), but one caught my eye this morning by Charlie Schluting over on Enterprise Networking Planet.

Two things struck me interesting about Charlie’s post.

First, he noticed the changes in the UI we’ve been slowly making over the last few releases. If you’ve ever done UI design, you know how much sweat goes into every little detail, and how much momentum a design carries over time. That someone noticed the new changes *and* liked them is a HUGE win for the…

Below are a few easter egg features found inside Splunk.

• From the commandline: “splunk ftw” produces an ascii-art “O’Rly?“.
• From the commandline: the “outputrawr” produces ascii-art fireworks.
• From the searchbox, piping results to the “marklar” processor (e.g. “*|marklar”), converts all search results into the Marklarian language.
• From the searchbox, piping result to the “loglady” processor (e.g., “*|loglady”), converts all the search results into quotes from Twin Peaks’s LogLady.

Enjoy them while they last, before they are removed by the Silliness Police, who%$($%%$
^H^H^H^NO CARRIER

A few months ago I wrote about our effort to automate and open up product planning by implementing a process around distilling product inputs into requirements using Jira in support of an agile/scrum based development model. I’ve rarely had so much response to a post… dozens of product managers at companies large and small wrote me and commented about their own efforts along the same lines. Many asked for our specs on our Jira customizations.

We were at the beginning of this effort when I wrote that post. In the intervening 3+ months we’ve completed the first round of Jira customizations (thanks to lots of help from Dave Pickering and the team at New Aspects of Software, a fantastic consulting firm specializing…

Most of the time logs contain data which, by humans, can be easily recognized as either completely or semi-structured information. Being able to extract structure in log data is a necessary first step to further, more interesting, analysis. While it would be great to be able to automatically extract the structure from all log data, splunk cannot rival the brain’s performance at this time, however it is able to tap into your brain for help Read on ……

Problem definition:

Extract structured information (in the form of key/field=value form) from un/semi-structured log data. Note: for the purpose of this post key or field are used interchangeably to denote a variable name.

Problem examples:

Splunk debug message (humans: easy, machine: easy)

12-03-2007 13:51:55.114 DEBUG SearchPipelinePerformance - processor=save queryid=_1196718714_619358 executetime=0.014secs
ideal…

Keeping track of new browser releases these days can be really challenging. It is less than ideal if your payment processor is throwing a JavaScript onsubmit exception effectively canceling all transactions.

Here is a little technique for indexing JavaScript exceptions in your production and development environments using Splunk.

In JavaScript create an onerror event handler that makes an HTTP request to a server that has access logs indexed by Splunk.

    function JSErrorLogger(httpBeacon){
        var self = this;
        self.handler = function(msg, url, line){
            var log = {
                "date":new Date(),
                "type":"jserror",
                "line":line,
                "msg":msg,
                "url":url
            }
            var logStr = "";
            for(var i in log){
                logStr += i + ":" + log[i] + " ";
            }
            var imgObj = new Image();
            imgObj.src = httpBeacon + "?" + logStr;…