Blogs: Dev

Splunkbase, a project near and dear to my heart, has re-launched on splunkbase.com.  Behind the scenes, Splunkbase has been running all along, as it is what handles app browsing and installation in the product, but we’re thrilled to open the site once again to the community, as the redeployment of Splunkbase was requested by Splunkers around the world.  For those folks who have never visited the site, it’s our interface to share Splunk Apps you’ve built with the community, and to download apps built by fellow community members, Splunk partners, and Splunk engineers here at HQ.

We’ve given a facelift to most areas of the site, but we’ve kept the previous 3.x UI active under the Archive tab, where you’ll you can download…

Data can reside anywhere and Splunk recognizes that fact by providing the concept of forwarders. The Splunk Forwarder will collect data locally and send it to a central Splunk indexer which may reside in a remote location. One of the great advantages of this approach is that forwarders maintain an internal index for where they left off when sending data. If for some reason the Splunk Indexer has to be taken offline, the forwarder can resume its task after the indexer is brought back up. Another advantage to forwarders is that they can load balance delivery to multiple indexers. Even a Splunk Light Forwarder (a forwarder that consumes minimal CPU resources and network bandwidth) can participate in an auto load…

Introducing SplunkMSE (Splunk MySQL Storage Engine).

SQL is the lingua franca of structured data.  Likewise, Splunk is the way to work with highly unstructured data generated in the data center.  Data residing in relational databases can be analyzed via a plethora of off the shelf tools like Excel, Tableau, Cognos, Crystal Reports and on and on.   SQL is well known by developers everywhere. What better idea than using these tools to work with data that lives within Splunk?

SplunkMSE is fully open source. Visit SplunkMSE’s home site  for downloads, installation instructions, detailed documentation, source code and more. While there, I encourage you to ask questions, file bugs and if the overwhelming urge to fix them should arise, feel free to do so.

To see…

This small, unofficial project integrates an open-source AIM (AOL Instant Messaging) Chatbot with Splunk 4, allowing ad hoc searching, running of prepared searches, and real-time search alerting via instant messaging.

What’s real-time searching? It’s new in Splunk 4.1, out shortly, and will allow users to search for “real-time” events, within seconds of them reaching Splunk. Most usefully, you can set up real-time searches and be IM’d with the matching events the second they show up. You could ask to be IM’d, for example, whenever someone logs into your system, whenever there’s an error, whenever someone logs in as root, etc.

Above is a screen capture of real-time alerts printing out for each time someone downloads Splunk!

Note: You can use this project with Splunk…

Well, a few months ago, a couple of the support folks asked me about the feasibility of creating cases in Salesforce through the command line in Python, and after doing a bit of research, I realized (surprisingly) that there really wasn’t a good way to use the Salesforce API in Python *if* you want to parse a WSDL (There is an excellent package called Beatbox if you don’t want or need to parse the XML). There is a very good Python SOAP client out these days called Suds, which unlike ZSI and SOAPpy is able to handle SOAP namespace prefixes (a key for handling Salesforce’s WSDLs), but it doesn’t handle the nitty-gritty details of the SOAP interaction for you (for instance,…

“Serendipity is looking in a haystack for a needle and discovering a farmer’s daughter” – Julius Comroe

I just read the quote in a presentation from Matt Jones of BERG at the DXf conference. There is so much i love about this presentation i don’t know where to start. Just click through it ( embedded below ) and have your own reaction. It’s clearly designed to be a fun/light read. I think I clicked at about one slide per few seconds. Then went back and stopped on a few that really spoke to me. It was entertainment that made me think which then made me smile.

At its heart, splunk is a time machine. It allows someone to go back in time and…

Every week i run into someone that is having performance issues and they are not aware you can just add another server or two or ten. I’ll travel to meet a company and I’ll ask how many servers they are using for Splunk to search/index/report on a terabyte a day. They will say a couple. I’ll then ask how many they have for a similar sized hadoop or data warehouse project. They will say 50 to 100X that number. Look if your going to give these systems 300+ servers, can we please get 15?

Somehow there is a breakdown in our communication that we scale like all other good architectures.

The following are hopefully some easy pictures to help tell the story.…

How people use Splunk is often a surprise to us – at least they are going beyond our original intent. Initially we thought of splunk as a search engine for log files, Google for your logs if you will, to help IT folks troubleshoot their complex systems. Quickly we found that users started Splunking config files, network packets, source code, email, etc. Over the years our customers have been dragging us into all sorts of new uses-cases like global windmill power plant data analysis, protein structure prediction, or just something simple like analyzing user behavior on a website.

Back several months, before the launch of 4.0, we were confronting at all the work ahead. As always, we had to make hard decisions about what is in and what is out. In 4.0 we had re-implemented much of the UI and a good chunks of the backend. With over 1000 paying customers and looking at a potentially challenging upgrade process and a huge testing task we needed to reduce risk to the schedule and product quality. It was a hard decision but we reduced the GA risk by pulling out the Free product until we GA’d and fixed most of the critical bugs. Our guess was that it would take 45-90 beyond the GA to get few maintenance releases out before we could test the free product.

There’s a big reason I haven’t blogged here for a while: Splunk 4. I’ve been so wrapped up in it for the last year that I haven’t really been interested in writing about anything else. Well, now it’s out, so I’m back! So I’ll kick it off with some background on why 4 is the Splunk I’ve always wanted and a little story about how my team and I have used Splunk ourselves in a new way the past few days.

The aspect of Splunk 4 that I’m most excited about is all of the ways that it makes IT data accessible to everyone, regardless of their job.

I’ve been a data fanatic since I started my first software company job 17…