**** UPDATE - 08/27/08 ****
I have updated the app with a few fixes found in the field.

• hopefully fixed issue on AIX (IBM jvm )
• added output of host/vm name on update messages. It was hard to tell where the messages were coming from
• added more debugging infor on startup to help debug connection issues.

Things that are still under-investigation.

• Pointing at lots of ESX servers and not VC. Seems as though some data is not coming back from ESX.
• Making work with older jvm’s ( currently it seems i require 1.5)

**** Original Post 08/10/08 ****
I’ve wanted to release this a few months ago but the project keeps getting stuck on the back-burner. Finally I’ve cleaned it up and had a few people try it and it seems to work well. I’m sure there are configurations and versions out there that will have issues - please write me back ( my first name at splunk.com ) if it does not work as advertised.

Everyone has their favorite customer.
I have one too and he is the CTO of a very cool IVR/VoIP platform. His name is RJ Auburn

Around here is synonomys with filing 34 bugs between sunday 9PM when we push bits to the site and 9AM when we get in to the office. I dont mean the usual the UI-is-off-by-10-pixels but complex indexing or distributed search bugs. Well, sometimes is its a trivial thing we missed, but usually he is usually pushing splunk to its limits. Its not often that a CTO and “industry expert” is the one to personally put splunk through its paces - but it’s RJ is like that and gets his hands dirty - and splunk is the better for it.

RJ and Voxeo are one of a few, but quickly growing, number of companies that are using splunk in a multi-tenant environment. This means using splunk to to collect data across multiple tenants in a hosted environment and then using splunk for searching and reporting on a per customer basis. Often the output of the searches/reports is rendered for the customer do they can see what is going on within the service. Customer dashboards and activity reports are a common usecase for splunk. Below are some of the images from the voxeo service:

We here at splunk are into processing lots of data. Our external marketing focuses mostly on hardcore IT data but internally we play with all sorts of data sets : government stats, sports stats, even music as shown by Brian cool post.

I just wanted to congratulate Nathan over at FlowingData for crossing the 3100 subscriber mark.

FlowingData is a fantastic example of the hidden value in the data all around us. As more and more of what we do is documented by computers the impact of statistics has become less of a hard-core math geek sport and more within the reach of anyone’s curiosity. His daily posts are a constant reminder of how statistics has become a crossover genre.

Thank you Nathan!
e

I’m looking for some help.
I’ve built a VMWare app for splunk and in the process of doing the same for Xen. These Apps use the VMWare and Xensource API’s to index everything about the VM environment. When combined with splunk instances running within the guest OS you get a very comprehensive historical picture. I’m curious are there any splunk customers out there using VMWare or Xen? I’m looking for usecases so that i better understand how to configure the apps. I’d be curious to know what types of information would be useful to capture and what types of searches would one want to perform. Both Xen and VMWare have so much data available that configuration could be complicated. I’m trying to narrow it down to several useful out of the box configurations. If your have any thoughts comment here or email me at erik at splunk dot com.

Thanks
e.

The following is copped from a support email by Stephen Sorkin who is the man behind the splunk server curtain … thought it should go broader.

I’m the manager of the search and indexing team at Splunk. We’re still in the process of writing up our findings from storage benchmarks but here are the general details.

High IO/s typically means both faster indexing in general and faster searching of rare, temporally incoherent events. On average, we’ve seen indexing speeds increase by about 66% going from an 7200 RPM SATA RAID to a 15K RPM SCSI RAID. We’ve seen comparable performance from SCSI and SAS RAIDs, provided they’re 15K RPM.

The best best benchmarking tool we’ve found for measuring how Splunk will behave on your disk hardware is bonnie++. If your disk subsystem can sustain 800 IO/s, you’re in good shape.

Hey all,

It’s taken longer than we would have liked but our 3rd preview build has been posted.
Get’um here

A bunch of work has gone into windows stability, tons of bugs were fixed, and a bunch of customer requests have been implemented ( we will let you know out of band ). We expect that this release should be more stable, slightly faster, and less buggy.

Left to do, we still have a bunch of IE work, performance improvements, and cleaning up of some features like interactive field extraction and event type discovery.

Its still not production ready so don’t even think of trying it out for real - and there is no guarantee that migration will work from a preview to GA ( we will migrate from 3.1.x to GA but not preview ). Also, don’t run splunk as root - its just not good to do until we run through all our testing.

As always, please send us feedback at splunkpreview@splunk.com or hit us up on IRC (irc.efnet.org #splunk).
The last round of info from Preview #2 was awesome please keep it up!

e.

Happy new year (bit early) all dev.splunk.com readers….
We have just posted our second 3.2 preview release. (build number 30455)

Its packed with holiday goodness, albeit very raw.

First you will notice we have posted a windows build. Its been in the cooker since last Feb and thanks to Mitch, Ledio, Igor and a bit of Amrit we now have a single code base that rocks on linux, mac, solaris, freebsd, aix, AND windows. This was not an easy feat as evidenced by our gift of a pony (soft and electronic) to Mitch for his effort. Its still very raw (the build not the pony), and has a tendency to crash because of a memory fragentation and limited vm space. Which will be fixed by GA… MarkB. will post more on the build so stay tuned for details. Its a big deal for us so be patient and we sure could use feedback on how to make it the best it can be.

Also in this release you will see the UI starts to get some of the async search results. Over the next few releases we will be moving to fully async search in the UI. It will take a few turns but this preview has some of the first cut.

Splunk fans.

We have posted the our first of many preview releases. You can find them here:
Our hope is that every week or two as new features or API’s become usable that we post builds soliciting feedback.
This first post has a bunch of backend and UI performance improvements as well as some new but hidden features:

• live searching of data
• flexible roles
• scripted authentication
• event decoration ( for the xmas season )
• auditing of splunk server actions
• file system change detection
• improved (proper) sub second support
• transaction search
• new experimental simple search interface
• “where” support in search clause ( you dont need to use the “| where” anymore and can just search for foo=10 )

I’m not going to explain here what these things mean or how to find them or use them
Instead the product managers and developers will post here with ideas on what to try and what feedback we are looking for.

I’d like to thank in advance those brave few of you that have the few minutes to install these builds and give us your feedback.

e.

Hi all,

Just a heads up that we are moving to a model where we post previews of upcoming releases.

Starting now, we are going into a mode where long before a GA release we will be posting development builds. At first, they may be a few weeks apart but over time our goal is to post builds as soon as new functionality or API’s are ready for comment.

This first Preview #1 will have backend performance and scale improvements as well as some cool new features. The developers and PM’s will be posting to this blog the specifics of what is new, how to try it, and where we are going.

Our hope is that we get early feedback on new features and API’s before we actually ship.

Thanks in advance for helping try out our early wares.

Kinds Regards,

e.

I find this hard to explain even though its an extremely simple concept. It would be nice to get some feedback since I think we want to productize the idea but we are not clear on what makes sense.

If I have a search/report that I want to run faster, I will save that search and have splunk run it over a small timeframe (5,15,30,60 min) taking the results of that search/report and feeding them back into an index i create to hold cached results.

For example, suppose I like to run nightly reports where I show “top users by bandwidth”. Its easy enough to run the report every night, but suppose there are times during the day when I want incrementals, or I want to look at last week, or perhaps get dailies over a month. Every time I run the search/report I need to search and recalculate “top users by bandwidth”, which if over billions of events can take time

Instead, I’ll just save the search/report and have Splunk run it every 15 minutes with the results being sent to a “cache” index. This way if I ever want to do an adhoc search on “top users” or if I want to do “weekly reports by day” all the data is precalculated.