Tim Jones of Agora Games posted a good summary of his experience with Splunk. Tim reveals what we’ve known for some time. Splunk is incredibly flexible and powerful but sometimes finding the Splunk documentation to do exactly what you want isn’t as easy as it should be.

We’ve struggled over the years to keeping our documentation both up to date and easy to use. Earlier this year we moved to a wiki based approach to Splunk documentation in hopes of keeping it more up to date and usable with inter-documentation links. Suffice to say we are still embryonic in our use of wiki technology as applied to documentation. We power our docs site with MediaWiki the PHP wiki technology that runs Wikipedia. Along the way we’ve had to add a lot of capability around the MediaWiki platform to control docs permissions and versioning.

If you sign-up as a Splunk Community member you can modify and add to the Splunk Knowledgebase and docs wiki yourself including:

• edit discussion tabs
• edit any page except for major landing pages and
• add new pages.

We’re taking this “extended community approach” to documentation because we know there are many people like Tim that have a the ability to help us make not just the Splunk download and bits better, but also the Splunk documentation better and more complete. We realize the risk in opening up our documentation to the community is that things won’t always be as easy to find as they should. But we believe in the long run this social approach to documentation will ultimately make Splunk a much better experience.

Please let us know what your think and how we can improve.

Happy Splunking

The past couple of days I’ve been visiting China meeting with some of our technology and channel partners. It just so happens I was present in Beijing for the 20th anniversary of the 1989 Tiananmen Square Events. Yes it really did happen despite what the Chinese government says. Speaking on Saturday at the F5 APAC Sales Kickoff I found myself staying over the weekend with Sunday off to roam around Beijing like a tourist, something I rarely get a chance to do on business trips. It is amazing to me to see how the Chinese and Taiwanese work on Saturdays. In the US we rarely see that. Europeans chastise Americans for working too hard but I guess they should really see the work ethic in Asia and then we’d look more normal.

Watching the 2008 Beijing Olympics last summer things there certainly seemed more normal than 20 years ago, but being there in person with all the festivities gone things seemed really strange to me. It is very difficult to describe. Maybe I was jaded by all the newspapers I’d read on the way to Beijing. On a nice long 13 hour flight from Washington DC with plenty of reading material I consumed James Kynge’s piece in the Financial Times questioning whether the Western media really understood why the student demonstrators were protesting. He went on ascribing the word “democracy” with the student motivations and questioning whether we or they really knew what it meant despite the fact that he spells out their desires in plan old English which sounds like democracy to me.

“Almost everything fell within its scope: campaigns against corruption, nepotism, inflation, police brutality, bureaucracy, official privilege, media censorship, human rights abuses, cramped student dormitories and the smothering of democratic urges. But to say the demonstrations were to “demand democracy” is an oversimplification.”
James Kynge, Financial Times

It’s almost impossible to describe the strange feeling I got while walking through Tiananmen Square observing the soldiers and the huge portrait of General Mao that dominates the landscape. Maybe part of it was due to the increased tension of the anniversary. Maybe not. Tiananmen has come to symbolize the unspoken and largely unrecognized tension between the economic progress driving modern China and the old fashion communist government still ruling there. The Chinese seem to have a foot in both camps. The eeriness I felt came not only from my surroundings and an understanding of the principles they stood for but also from the reaction of my Chinese and Taiwanese friends. Their usually jubilant outgoing personalities were completely subdued in the square. Was a sign of respect and mourning that drove their thoughts? Perhaps to some extent. But in quiet whispers and conversations out of the ear shot of any “green” uniformed soldiers (versus the “blue uniformed” security guards they confessed to being actually scared to speak for fear of someone or something listening. Challenging them I said, “surely you must be joking.” But it was no joke. Only when we crossed the street into the forbidden city did their usual personalities return.

Of course this began a prolonged conversation over the next 24 hours as we visited the great wall, a new Beijing restaurant and departed through the impressive new Beijing airport. I kept asking and trying to understand. How can a country of so many people be controlled by the minds of so few? What are the real limitations to speak out? And what effect will economic progress have on the political future of China? There was no shortage of stories supporting the fact that the government still does take a very heavy hand to those who disagree. But rather than discuss it, everyday Beijing seems to sweep the event of 20 years ago under the rug. As one of my Chinese friends said, “everyone is embarrassed and we just pretend it never happened.”

At the same time I was traveling through out China, the articles started pouring in about Beijing’s efforts to step up Internet and IT censorship. Upon reading the perspectives pouring in about “Green Dam” I was reminded of the impact the technology industry is having on the whole situation. It was bad enough I couldn’t get to sites like Twitter and Youtube form my hotel room. Now the Chinese government is requiring every PC sold in the country starting July 1st has to have special software blocking all sorts of things. The move is being presented as an attempt to protect children from online pornography but is obviously one more attempt by Beijing take its censorship to a new level. China currently has the world’s most sophisticated and multi-layered system of Internet censorship. Objectionable content on domestic Web sites is deleted or prevented from being published, and access to a large number of overseas Web sites is blocked or “filtered.” Decisions about what to censor are based on the Chinese government’s attempts to control the minds of 1.2B Chinese. There is no transparency or accountability, no public consultation in developing block lists or censorship criteria, and no way to appeal the blockage or removal of Web content.

In a notice to PC makers, the Ministry of Industry and Information Technology said all PCs shipped in China needed to offer Green Dam/Youth Escort, identified as a “green internet filtering software”, either pre-installed or as part of basic software packages. In May 2008, the government picked Jinhui Technology and Dazheng Language Technology, two Chinese software companies to develop the software, according to a contract award notice from the MIIT. While these companies claim their software is only being used to block sites although last year, researchers discovered that a Chinese version of Skype contained the ability to block politically sensitive words in instant messaging chats, and to keep a record of the use of such words.

It’s kinda a funny story and although it seems so long ago it was just 18 months ago. I was traveling in Europe starting to talk with potential customers who had downloaded and installed Splunk (3.0 variety). My very first meeting was with a guy name Scott Davies VP of E-commerce Trading Platforms at Royal Bank of Scottland in London’s Bishop Gate. I had the opening slide to our presentation up when Scott walked in the room. He was very polite, asked us if we wanted some still or sparkling water and wanted to know how our trip was progressing thus far. Finished with the pleasantries he than quipped, “I love your product, but when are you going to change your name.”

Seems “Splunk” didn’t quite translate all that well in the UK. Although Colin Barker and Steven Arnold didn’t seem to mind. Fast forward to October 2008 and here we are with more than 60 customers in Europe including several major banks, telecommunication providers and large enterprises. And now we have a big shot head of EMEA and an incredible team on the ground in London. Welcome Brian Haynes!

I first met Brian about three months ago at the Berkeley Hotel in London. We hit it off immediately. Brian was incredibly excited about our free download model as he had experienced similar success with companies like Legato that initially followed a simlar model. The difference he said was, “Splunk really believes in fostering a global community of users around its product, something Legato never had.” As our new Vice President Sales for EMEA, Brian will no doubt help us really accelerate our growth in the European market. He joins us at a great time. Last week we attended the IP 08 show and our booth was mobbed with folks anxious to learn how they can Splunk their infrastructures.

As the global economy continues to crumble its amazing to see that we’re able to keep bringing value to customers around the world and grow our user and customer base by helping IT organizations do a lot more with less. The notion of a single universal platform that breaks down the silos between operations, security and compliance will certainly continue to thrive.

We’ve been hearing a lot lately about the death of SIEM technologies. But isn’t the question less about a legacy technology dying and more about the dimensions on which the next mass adopted security capability will be born? Clayton Christensen first described a model for disruptive technology in his book The Innovator’s Dilemma and his follow on The Innovator’s Solution. Christensen describes a theory about how disruptive technologies over take sustaining technologies by delivering value on new dimensions that established vendors overlook as unimportant, low end or just don’t think about because they’re too busy improving their legacy. Christensen’s work offers an interest framework to think about what’s taking place in the market for SIEM security management solutions.

Any enterprise trying to secure their IT infrastructures knows the state of the art in SIEM security approaches falls short. And trends like virtualization are making things even more difficult. System and security administrators and analysts are inundated with too many potential incidents and its too difficult and time consuming to investigate even a fraction of them. Achieving a greater comprehension of the meaning of potential incidents and the projection of their status in the near future is the real goal. The idea, called “situational awareness” is often, however, impossible to achieve. We are so dependent on pre-programed rules in our SIEM solutions that we lack the ability to perform our own analysis because the original raw data has been filtered out, thrown away or we have no practical way to make sense of it.

Observation: If the technology is sufficiently complex as to allow the vulnerability to exist, can we really build complex technology to catch all the possible issues or scenarios?

As a reference point see David Hazekamp, Security Architect at Motorola, talk about the importance of retaining all security data across the Motorola global SOC infrastructure and integrating access to all this data into existing SIEM solutions.

Of course reaching this understanding requires one suspends their disbelief about the effectiveness of current SIEM security technologies. Usually this means you’re not a vendor or you’re a vendor with little or no vested interest in current approaches. So with this let’s examine the typical enterprise deployment of security technologies.

Defense in Depth

This is where every good enterprise security architecture starts. In order to begin securing your environment you’ve got to have data, raw data. In most data centers this takes the form of syslog from network devices and servers, SNMP traps, OPSEC or LEA interfaces for firewall events, WMI for Windows desktop and server events, IDS and IPS signature scans and application level firewall examination of common services like FTP, HTTP, SFTP, SCP etc. The thinking is you need to look at everything. Perhaps you’ll even want to pull in information from physical security systems like badge readers.

Security Information Management (SIM)

The next step in the process is to manage all this raw data and filter it down to a manageable number of events, traps and alerts. Collecting, storing and providing some basic analysis on all this data is the job of a SIM. Typically, as Raffy points out, the data is parsed, normalized and stored in a structured RDBMS. Parsing, normalizing and structuring all this data is great if the data doesn’t change or you don’t have too much of it. But if you’re dealing with data formats that aren’t static or you’re trying to store terabytes of this data an RDBMS won’t be your friend.

Security Event Management (SEM)

Once a SIM has done it’s job you’re ready to aggregate, correlate and start reporting on potential incidents using a SEM to do the job. SEM’s usually consist of lots of rules that look for combination and patterns of events indicating that a possible attack or breach may be underway. Essentially the SEM rules attempt to codify what we humans know about vulnerabilities in our IT systems and possible ways to exploit them. The goal is to provide some real-time information usually in the form of reports, dashboards and visualizations to operations and security analysts who work to keep the infrastructure secure.

Situational Awareness (SA)

SIEM correlation can be interesting for discovering a pattern or related event but the ability to work an issue outside of these “canned” rules and events becomes the real problem. Unfortunately, what all to often happens is there are so many possible attacks, operations and security staff are overwhelmed with potential incidents to investigate and not every event or pattern of interest is going to be discovered via the pre-built rules. Situational awareness is the attempt to perceive environmental elements within a volume of space and time. Comprehension cannot be achieved if the data being bubbled up is filtered according to a set of rules and the technology does not allow a human to perform their own analysis of the raw data as generated by the environment itself. All technologies have their weaknesses and those that perform correlation are no different.

Thus whilst canned SIEM correlation provides value in bubbling things up — we still need the ability to dig into the raw data to fully perceive and comprehend what is taking place. Now mind us all SA is not a new concept. It has been applied rather robustly by decision-makers in complex, dynamic areas from aviation, air traffic control, power plant operations, military command and control — to more ordinary but nevertheless complex tasks such as driving an automobile or motorcycle. And yes it has been mentioned before in security operations, particularly in government agencies.

Without a doubt the past week has been the most amazing week in Splunk history. The crazy coast to coast multi-city launch left us all exhausted and electrified. A few of the things that stick in my mind…

First Splunk 3.2 including Splunk for Windows went live on our download page last Saturday and more than 40% of our downloads in the past week have been for our new Windows version. Then Nick Selby of 451 Group wrote an analyst brief on us. He said, “Splunk is awesome: it’s multiplatform, easy to install and easy to use. And with an abstraction layer of logs, configuration files and system messages, traps and alerts, it’s seriously useful.” 451 has a reputation for ripping vendors, so we’re flattered.

Dana Gardner, analyst with Interarbor wrote a very eloquent analysis of our platform launch on ZD Net. “Splunk has created the means to offer developers easy access to that data and the powerful inferences gleaned from comprehensive IT search. That means the data can go places no log file has gone before,” says Dana. Developers are certainly doing some way cool things with Splunk.

I’ve seen a couple of neat visualization applications including this one called Replay. It shows you a live or time lapsed view of your event streams. Here you can see the replay application hooked up to our internal wiki showing who’s doing what over a 24 hour period. Click on the image for the movie.

As for our own applications, the Splunk for PCI app drew tremendous interest at our series of Splunk Live events this past week. It’s just one example of how a business person with domain knowledge can package their own Splunk configuration as an application. If you haven’t seen Raffy’s video on the PCI Application, check it out here.

We also showed the Splunk for Change Management application as well. Seeing someone touch a file and watching the Splunk dashboard update instantaneously is an awesome display of how flexible Splunk has become. Check out the developer program for yourself and get your goods up on SplunkBase so we can all check em out.

Recently, Johnvey Hwang wrote a post called Standing on Our Own Platform. He was the first one at Splunk to break the ice and use the “P” word. Now it’s out there. What do we see when we stand on our own platform? While only you and the future will tell us — there are a few things we hope to see on the horizon.

First, it’s our belief there’s a lot of money out there wasted on point products for managing networks, servers, applications … even security. A lot of these systems redundantly collect, transmit and store much of the same machine generated data. Think of the network, storage and administration resources duplicated on all this stuff. By providing a platform where the same IT data can be managed once, resources can be freed for other projects.

Second, none of these products work together. If you’re running a network manager to collect and look at SNMP and netflow data you know it doesn’t integrate with your log management system and of course neither talks to your SIEM, SOA, virtualization or application framework monitoring consoles. Building a dense index of data from all of these tools enables correlation across all your silos of instrumentation.

Third, and perhaps most important, isn’t it frustrating to spend so much time getting a new tool running only to discover, it doesn’t do what you need? Allowing, as Johnvey calls it the “intrepid” sysadmin or the creative developer to build on top of our IT Search engine means you can make Splunk do exactly what you want and share it with others if you so desire.

We’re not just jumping on the bandwagon here. Sure everyone seems to have a platform play. It feels like Web 3.0. Google has the mobile phone thing. Facebook, MySpace and Ning have social networking. Salesforce.com has AppExchange and force.com. For interesting reading on the phenomenon check out Marc Andreessen’s post from a few months ago on the topic.

Everyone here hopes to convince you that the thoughtfulness by which we’re going about this will yield much more than a bunch of hype. Ultimately the goal is to allow anyone to unleash their creativity to devise their own way to use Splunk.

Much more to come for sure. If you have thoughts or want to get involved — let us know anytime.

The US economy is heading into a recession and technology spending is in for a steep decline in 2008. So every major prognosticator and news outlet from the Wall Street Journal to the Financial Times would have us believe.

Are these people watching the same movie I am? There are two problems I have with this economic hyperbole. Yes that’s what it is. I guess it sells newspapers and gets people to watch things like CNBC. But boy is it misleading.

First of all, in macroeconomics, a recession is a decline in any country’s gross domestic product (GDP), or negative real economic growth, for two or more successive quarters of a year. Yet nobody that I’ve read is forecasting negative growth. They’re forecasting a potential slow down in growth from the current 3.5% per quarter to 1.5 to 2.5% per quarter. But the news outlets feel compelled to use the “R” word just to get attention. Totally irresponsible.

On to my second gripe. With regards to technology and IT spending, I believe, based on what I see, we are in beginning of a long-term gradual increase in IT spending within large enterprises that started eighteen to twenty four months ago.

Sure the current credit crisis may have a short-term impact on budgets within Financial Services companies, but I don’t see any slow down yet. The major consumer, commercial and investment banks we work with have so many critical, revenue generating IT projects in backlog I fail to see how spending is going to slow at all. The telecommunication sector is finally back on the mend after the post early 2000’s bubble and hangover.

Social media, online shopping and the always on dimension of the Internet have online services and large Internet sites like MySpace and Amazon accelerating software, hardware and services spending just to keep up. And security, privacy and compliance initiatives and mandates have companies, service providers and government agencies increasing spending on these items by some 20% or more in 2008 to try and limit their exposure and risk.

Just a month ago the Financial Times had a great piece entitled “What’s on CIO wishlists?” Here’s a quick summary.

1. Business alignment and strategy
2. Hiring and retaining the best staff
3. IT innovation/new methodologies
4. Security
5. Collaboration technologies
6. Controlling costs
7. Compliance and regulation
8. Virtualisation
9. Customer service
10. Mobility (Green issues came 11th)

Doesn’t look like a slow down to me.

I’m not sure if it’s the start of a new quarter, the full moon or my two seven year old boys that have me thinking about this, but we seem to be blowing a lot of things up lately. A few examples…

1. We blew up our product development process
2. We blew up lots of our software
3. We blew up our business planning process

When I say we “blew ________ up” (enter your own thing here) I mean we decided to take another course of action, look in the other direction, put other people in charge or just plain start over from scratch. Combustibles are exciting for lots of reasons (especially to second graders) but as a new type of business tool?

I’ve written in previous posts about our move to an Agile product development process. This required us to literally discharge our old way of taking input from customers, scoping features, planning releases and testing. Of course it also meant we had to ignite our underlying work flow and tools supporting product development. It all made me a tad nervous : { For more than a month I couldn’t tell you what would appear in our next release or when the release might be available for download. If you use Splunk, you know that we live and die by our product road map and release schedule. During that month our engineering, qa and product management teams went through a metamorphoses. They moved from being top down, planning driven to bottom up, innovation driven. We had reached the point where we couldn’t plan or prioritize features. The old process of having a team set out a plan and working towards a release wasn’t working anymore. So we blew it up. Now we have a process where by parallel scrum teams work on various facets of the product and they do the planning, constantly. It’s interesting how nobody, but yet everybody is in charge. The initial results are just in. Splunk 3.1 will soon be available for download in a mere eight weeks after Splunk 3.0 was posted. And Splunk 3.2 will be released in beta eight weeks from now. That may not sound like much but when you look at the amount of innovation in each release, the speed with which we’re moving enhancement requests from the field into features and the improved quality of each release it appears remarkable from where I stand.

Detonating software is always dangerous. Will it ever come back together again? Were we right about the surface area becoming too large or the architecture verging on too complex? Stay tuned. We’re in the process of blowing up a lot of our software. For example, we’ve realized our past approach to administration just doesn’t scale. Early on we built a nice UI for editing lots of the configuration properties of a Splunk server. But over time our ability to quickly add features outstripped the surface area of the UI. So we’ve been making configuration parameters available in editable configuration files. Now that is all fine and good but it’s not very discoverable and it’s completely out of context with the task at hand when you’re using the product. Definitely a candidate for explosives. Sometime in the near future you’ll see the administrative side of Splunk blasted for a much more scalable, discoverable and in context design we call “search based administration.” This is one small example of how we’re constantly blowing up our software.

Recently we’ve also been lighting the fuse on our business planning process. It used to be we’d have a few days at the beginning of our quarter when each department in the company (sales, marketing, engineering, customer support etc) would get together and have their own planning process. As we’ve doubled in size since the beginning of the year our old way of planning wasn’t working. Despite our completely open work environment (we have no cubicles or offices) communication across groups had slowed to the point where it was causing a lack of effective planning. You guessed it. We blasted it. Started over. Asked everyone what would make for a better planning process. This quarter we started with a full day of conversations. Everyone was invited to run a one hour discussion forum on any topic they wanted. The only rule was you had to publish it a week a head of time and provide a brief description of the topic on our internal wiki. We had 15 discussion forums run by people all over the company. That was it. Our Q4 planning. A bunch of conversations. We’ll see how far it gets us ; )

BTW, I heard someone at Splunk say in response to blowing things up,

“perhaps companies that don’t blow things up often enough end up blowing up themselves.”

Certainly food for thought. I’m keeping my dynamite close by.

Last week Splunk sponsored ComputerWorld’s Infrastructure World conference along with HP and IBM. I needed to come up with a talk and I wanted to do something new.

I’ve been thinking about how to describe the challenges we have managing all this changing technology and innovation. Note this is seriously a work in progress. I’m developing a theory that there are three fundamental drivers to data center chaos.

• expectations,
• complexity and
• accountability

Any new business or consumer technology can be quickly met with significant expectations if it becomes successful. Our dependence on everything from wireless email, online travel reservation systems and hosted software as a service dramatically increases the expectations these technologies will always be available, fast and do everything we want. Examples of failed expectation are everywhere. A few examples. On June, 20th United Airlines canceled 24 flights and delayed another 286 flights due to a “computer gremlin.” Research in Motion recently experienced yet another 24 hour email outage and more than 2.5M users were without service in North America. Salesforce.com, pioneers of Software as a Service (SAAS), a more reliable alternative to running it yourself continue to have outages as well.

Rising expectations, success and dependency force increased complexity in both scope and scale to meet demand. Scope complexity abounds as more and more features and capabilities are added to the services we depend on. I used an example of Citigroup’s internal SOA architecture that has five federated ESBs — one of every technology flavor. Scale complexity occurs as infrastructures grow so large they begin to stress under their own weight. Salesforce.com for example is now processing more than 90M transactions a day through their web interface and AppExchange platform. At a meager 10 messages per transaction that’s almost a billion messages a day going through the infrastructure. Wow. Imagine finding a needle in that haystack.

Finally once popularity rises and the technology become established, accountability arrives. Now we have to worry how safe is the technology and in many cases monitor what people are doing with it. Everyone by now knows of the TJX situation where 45.7M credit and debit card numbers were stolen by hackers that somehow infiltrated its processing systems. The first card numbers were stolen three years ago and still there is no definitive explanation. Everything from cracked WEP keys, software tampered kiosks and insider job have been offered as possible causes. More recently TDAmeritrade and Monster.com have experienced similar breaches of user and account information totaling into the millions. And compliance is everywhere. SOX, PCI, ITIL, HIPAA, FFIEC, FISMA, ISO, CoBIT, COSO and other mandates means IT staff have reduced access and visibility into the systems their trying to manage and keep running.

expectations + complexity + accountability = chaos

I’m interested in your thoughts on the direction this is taking. I’ll be sure to blog more later as the ideas develop.

Who am I to second guess it. Virtualization is hot. In the past week VMWare went public closing at the end of the first day with a $20B market cap and Citrix agreed to buy XenSource for $500M.

WOW! This kind of activity make the bubble days pale in comparison.

I mean okay VMWare, hot company, fast top line revenue growth but also accelerating expenses. In 2006 the company reported revenues of $704M and net income of $87M or 12% of revenue. In 2005 VMWare reported $387M in revenues and net income of $67M or 17%. So revenue was up 82% but net income is declining on higher spending all around. If the company continues to grow revenues again this year at 82% the current $20B market cap means a 28x trailing twelve months revenue and a 15x current run rate revenues.

Compare VMWare to the Bladelogic IPO or the Opsware acquisition by HP and it looks pricey by comparison. But, given the market is so starved for growth stories it kinda makes sense.

But, XenSource is another story. The company just started shipping product in January of this year and according to Business Week’s Aaron Pressman and 451’s Rachel Chalmers, XenSource had less than $1M in revenue over the past year. That means Citrix paid 500x trailing 12 month revenues. WOW! Okay yeah Citrix needs a new game, they’ve been looking for their 2.0 story forever. But I mean anyone could pick up Xen and integrate it. It’s open source for crying out loud! Is there really $500M worth of value in the XenSource management tool for the Xen Hypervisor? Citrix seems to think so.

I think this points more to the continuing trend of acquiring hot technologists not so much technologies. Despite what Matt Assay writes Tim O’Reilly may just be right on with his assessment of Open Source companies eventually being bought by proprietary companies. These acquirers are not buying the software or the licensing model, but the people. The licensing model doesn’t matter. Customers pay for innovative ideas that solve problems easier, cheaper and faster and they’ll buy it in almost any form if it works better than what they’ve got.

Sure Cirtrix paid a whole lot. But they might be right about owning the brightest minds in virtualization software. If virtualization is the future and they captivate the biggest thinkings perhaps 500x is not too much?

What’s so great about agile software development? Sure engineers think its great. I think it helps them feel empowered. Product managers think its great too, but secretly I think they’re still trying to figure it out. Apparently Oracle thinks it’s great. The company just paid $495 million for Agile Software Corporation (Nasdaq: AGIL), representing a 14% premium.

As a user I just want to know, what’s in it for me?

We just shipped Splunk 3.0. During the past seven months I’ve heard agile, scrum, sprints — all the cool concepts that are part of this revolutionary framework to spur innovation, more efficient product development cycles and a tighter loop with customers. So why then did it take us half a year to release something? I mean we’re still a start-up after all.

Don’t get me wrong, Splunk 3.0 is fantastic. All kinds of amazing, ground breaking stuff. I’m running it on my OS X laptops and desktops to monitor ps, top, iostat, vmstat and it comes in very handy for figuring out why things crash by indexing everything in /var/log. It also now generates cool interactive reports and Flash graphs of who’s getting the most SPAM on my FC5 mail server running Sendmail and Dovecot IMAP front ended by my Ironport box. Look out mom I’m gonna charge you extra. Check out all the new Splunk 3.0 features and download your own free copy.

What I’m getting at is we had many of the latest greatest features available months ago. But they were unavailable to me and other end users. Turns out even though we’re developing software by living and breathing much of the agile manifesto, we’re still struggling with how to conceive of and adhere to an agile release cycle which incorporates more than just engineering. Sure if we wanted we could just deliver everyone half broken releases every few days without an installer and lousy or no documentation. However, our business thrives by satisfying users with a complete and easy to use product that is high performance and high quality. See IT people are much less curious and tinkering when it comes to solutions and tools than developers. They just want the stuff to work. Turns out the massively complex infrastructure they’re managing gives them enough to toy with.

Sometimes we get customers saying they can’t install releases more than two or three times a year, but I have a hunch with a technology that’s moving as fast as IT Search — we need to release new features faster. So we’re working on moving this agile, defactored architecture we have underneath the covers all the way to the surface. And in the process challenging ourselves to move the whole stack to faster release cycles. We get input through our live product roadmap every day and we’ve got innumerable parallel engineering scrums going on at any one time. But it’s the last mile of figuring out what pieces to release when and how to package them which always seems to slow things down.

So what can you expect to see from us in the future?

• Smaller units of features.
• More frequent releases.
• An active beta program.
• A well supported Splunk developer network.

We’re VERY interested in your comments on how we’re doing and feedback on what you’d like to see. Feel free to contact me or our support team with your thoughts.

Happy Splunking!

I’m Michael Baum. Welcome to my blog.

I hope to find time to write about some of my favorite topics including:

• Splunk and IT Search.
• Technology gadgets and software — the stuff we all like to use.
• Datacenter applications, servers, networks and security — the stuff we all have to keep running.
• Business, entrepreneurship and venture capital.
• Wall street and investing.

Comments are always welcome and you can also reach me via email at thebaum (at) splunk (dot) com.