On what was an incredibly beautiful day we had more than 100 Splunk devotees attend our first ever SplunkLive event in Seattle last week. In the shadow of Microsoft we talked about our Windows and Microsoft strategy and compare notes with lots of customers that are running mixed Microsoft, Linux, Solaris environments. Many of our customers with Microsoft Active Directory, Exchange and SharePoint environments are utilizing Splunk to troubleshoot problems and implement security and compliance controls in large-scale, distributed environments. But, I’m still surprised at how little Microsoft .NET we’re seeing in production large-scale applications.

Three Seattle-based customers presented their views on managing mission critical applications, IT data consolidation and Splunk.

• T-Mobile USA
• Blue Nile
• Washington State University

T-Mobile USA

Sean White, Senior Engineer with T-Mobile Operations in Bellevue talked with us about their global rollout of Splunk. Sean is a member of the security engineering team charged with incident response, IDS, vulnerability scanning, anti-virus and enterprise unified logging. He graduated with a B.S. in Computer Science from University of Kansas and has a deep background in large telecom environments initially as a system administrator and webmaster, SS7 network C&C and performance, engineering and now in information security. Sean has been at T-Mobile for 4 years, prior to that at Cingular, AT&T Wireless. T-Mobile USA is the 4th largest US national provider of wireless voice, messaging, and data services to 34M subscribers with annual revenues of $17B. T-Mobile USA is the US operating entity of T-Mobile International AG, the mobile communications subsidiary of Deutsche Telekom AG (NYSE: DT). Deutsche Telekom is one of the largest telecommunications companies in the world, with nearly 120 million customers worldwide

It all started with PCI Compliance

Like many of our enterprise customers, T-Mobile started working with Splunk in one area but quickly saw the value of expanding into others. For Sean and his team, PCI Compliance was the beginning of the Splunk solution footprint, but soon everyone realized the consolidation of logs, events, messages, configurations and changes meant a whole lot more.

Beginning with proving PCI compliance, T-Mobile has very specific requirements. PCI Section 10: Track and monitor all access to network resources handling cardholder data. But in T-Mobile’s case scale was a big issue. Fulfilling PCI DSS Section 10 meant tracking 26+ in-scope applications and the ability to trace transactions from start to finish across 650+ servers running Windows, Linux and Unix varieties. It also means more than 100 individuals logging into Splunk on a daily basis as part of the process.

The Splunk Set-up

The Splunk configuration consists of

• Pairs of forwarders set up in each of 4 geographic locations.
• Three short term indexers + 1 short term search box.
• Three Long-term search boxes hooked into a 32 TB NAS.
• Centrally controlled from a single deployment server.

The current installation is indexing more than 600GB/day of data and has just passed the 10B event mark. Controlling access to all this data is critical and T-Mobile has Splunk roles set up for managers and application teams to limit access to subsets of the data. The ability to segregate data access along lines of duties is critical to prove PCI compliance.

The Business Case for a SOC

In addition to proving PCI Compliance, T-Mobile has discovered Splunk’s use for Security as well. Not long ago, a SIEM vendor would have told you IDS and firewall logs were all you need. That >=2 sources of data == correlation. Not so much.

“All the best new vulnerabilities are coming in on the application layer.”
- Sean White

Enterprise logging—visibility into all of your IT data—is absolutely critical in defending against modern blended attacks. At T-Mobile Splunk has become a primary analysis tool for deciphering what is happening to the applications, servers and devices on the network. A few saved searches and Splunk helps does real correlation.

Nothing Boring about Logs and IT Data!

PCI Compliance mandates gave T-Mobile the excuse (read funding) to start an enterprise logging initiative. Logging all security, network and application events can truly give insight needed to not only measure and report on compliance controls but also to run a more secure and effective business. PCI has also discovered that integrating the ability to ask any question of their environment and get immediate answers also provides a pile of value to the help desk operations and better business intelligence functions.

“All the information about your company is in your logs—there’s nothing boring about it.”

Blue Nile

Jerry Brennock, Director Core Development at Blue Nile explained how the company is using Splunk to improve the experience of buying diamonds over the Web. Blue Nile, Inc. is an online retailer of diamonds and fine jewelry offering in-depth educational materials and unique online tools that place consumers in control of the jewelry shopping process. Importantly, the focus is on giving customers a great experience at a a great price – this translates to requiring high quality at a low cost. Jerry’s team team builds and support the infrastructure and applications for merchandising and marketing, including the website. He’s been with Blue Nile for 10 years and in the e-commerce space for more than 17.

The Killer Diamond App

Diamond Search is undoubtedly the killer application for Blue Nile’s E-commerce experience. It’s an asynchronous javascript app that has to work across any browser and there are many non-obvious use cases. All three of these factors means it is prone to failure in lots of edge cases.

“If this application isn’t fast and accurate, we don’t sell diamonds.”
- Jerry Brennock

Jerry’s team has embedded tracking pixels with name value pairs to track JavaScript profile information from each diamond search. This together with Web server 500 and 404 errors give the development, operations and customer support teams all the data they need to troubleshoot problems. The challenge is finding customer problems “in the moment” before the sale is lost.

Tim Jones of Agora Games posted a good summary of his experience with Splunk. Tim reveals what we’ve known for some time. Splunk is incredibly flexible and powerful but sometimes finding the Splunk documentation to do exactly what you want isn’t as easy as it should be.

We’ve struggled over the years to keeping our documentation both up to date and easy to use. Earlier this year we moved to a wiki based approach to Splunk documentation in hopes of keeping it more up to date and usable with inter-documentation links. Suffice to say we are still embryonic in our use of wiki technology as applied to documentation. We power our docs site with MediaWiki the PHP wiki technology that runs Wikipedia. Along the way we’ve had to add a lot of capability around the MediaWiki platform to control docs permissions and versioning.

If you sign-up as a Splunk Community member you can modify and add to the Splunk Knowledgebase and docs wiki yourself including:

• edit discussion tabs
• edit any page except for major landing pages and
• add new pages.

We’re taking this “extended community approach” to documentation because we know there are many people like Tim that have a the ability to help us make not just the Splunk download and bits better, but also the Splunk documentation better and more complete. We realize the risk in opening up our documentation to the community is that things won’t always be as easy to find as they should. But we believe in the long run this social approach to documentation will ultimately make Splunk a much better experience.

Please let us know what your think and how we can improve.

Happy Splunking

More than 300 people attended Splunk Live Taipei last week and our partners at Systex hosted an incredible show of Splunk use cases, customer speakers and hands-on labs. The Systex Splunk Lab provided attendees with the opportunity to use Splunk with CICS and IBM System z mainframe data, Windows, servers and desktops, Unix and Linux, customer service operations environments, telco provisioning environments and more.

I’ll be posting separately on the hands on the Systex Splunk Lab.

Our first guest customer speaker was Yi-Lang Tsai(蔡一郎) the Taiwan Chapter Chief Security Officer of the Global Honeynet Project and the Division Manager of the National Center for High-performance Computing, a Honeynet Project sponsor. Yi-Lang is also a freelance writer with more than 30 books published on operating systems, network and system security and IT management. He presented the very important botnet work Honeynet Project is doing and showed how his team is using Splunk to deepen their research and expose what they find to the Honeynet audience of security professionals worldwide.

What is Honeynet?

The mission of the Honeynet Project is to learn the tools, tactics, and motives of the blackhat community, and share the lessons learned. Honeynet is an all volunteer organization of security professionals around the world dedicated to researching cyber threats by deploying networks to be hacked. The goals are

• Awareness: to raise awareness of threats that exist,
• Information: for those already aware, tech and information about threats and
• Research: To give organizations the capabilities to learn more on their own.

Honeynet is completely open source and all of the work, research and findings are share. Everything captured is happening in the wild (there is no theory). The organization has no agenda, no employees and no product or service to sell.

Honey is simply a “high-interation” honeypot attracking any and all cyber threats and attacks. It is architecture, not a product or software that gets populated with live systems donated and run by the various Honeynet chapters globally.

Once the Honeynet is compromised, data is collected, correlated and analyzed to learn the tools, tactics, and motives of the blackhat community. Specific benefits to the global community of security professionals are the

Research : Identifying new tools and new tactics,

Profiling: Generating and maintaining lists of blackhats,

Protection: Early detection, warning and prediction,

Response: Forensics and incident response and

Self-defense.

Taiwan Honeynet Chapter’s Environment

Yi-Lang’s environment at the Taiwan National Center for High Performance Computing disitribuytes Honeynet/Honeypots to the Taiwan Education Network, Taiwan Chapter members and the GDH project. The environment makes heavy use of virtualization in its deployment, you might call it a “Virtual Machine Honeynet.” Its running on an advanced blade server with 128GB of memory running VMware ESX. The blade server uses either SAS OR SSD storage. More than 200 Windows 2K/2K3, Windows XP/Vista/7, Linux and FreeBSD servers run in high and low interaction honeypots.

The Taiwan Honeynet deployment is distributed across four different data centers in different geographies Taipei, Hsinchu, Taichung and Tainan. This distributed topology allows the honeypot to have a broad reaching capture network and makes use of idle network and CPU. This large-scale Honeynet deployment supports: Malware Collection and Analysis Honey-Driven Botnet Detection Client -Side Attack Malicious Web Server Exploring RFI Scripts Detection Fast-Flux Domain Service Tracking Research Alliance Distributed Search and Analysis on Honeynet Data

Why Splunk?

The Taiwan Honeynet teams uses Splunk to collect and manage information from the distributed Honeynet infrastructure including GBs of logs, 400k+ connections, 2GB+ of traffic flows and tools events and metrics.

http://blogs.splunk.com/thebaum/wp-content/uploads/2009/10/allindexdata.png

Data analysis is performed against a variety of pivot points that are automatically extracted from the Honeynet data sources. Date & Time, Malware Source IP address, Destination IP, Protocols, Files name and Malware MD5 are some of the main fields Splunk identifies and provides to the team for deeper analysis. In addition to Splunk searches and reports the team has built custom geo-dashboards with high resolution displays by tapping into the Splunk API.

This interactive geo-view provides the team Botnet detection, malware presence, Honeynet traffic flows and an instant status report all from one location.

Yong Sweah Liang (Linus), VP, Head of Infrastructure and Technology for Infocomm Asia Holdings Pte Ltd (IAHGames) was our second customer speaker.

IAH is an online game company operating some major properties including:

• EA SPORTS™ FIFA Online 2
• Granado Espada
• Dragonica
• Distribution of Box products
• BioShock®
• Grand Theft Auto IV

Wednesday and we’re at Splunk Live Princeton, NJ. What an awesome place. Princeton is home to a great university and some great culinary experiences. Check out Mediterra — an interesting mix of Italian and Spanish influences. Apparently it’s where all the Princeton parents treat their kids to dinner when they are in town. Next store to our venue was the great hope for the state of NJ — a new Governor. The current Governor has turned the state budget and tax base into toxic waste. Well things went much better for the more than 60 Splunk Live attendees in Princeton today, who gained insight into how a number of large Splunk customers keep their mission critical applications running in a time of IT budget slash and burn.

Matthew Stevens, Director Software Systems and Architecture at Comcast provides guidance to Comcast executives on mission critical media systems and strategic systems architecture. Comcast is the country’s largest provider of cable services serving 23.9 million cable customers, 15.3 million high-speed Internet customers and 7.0 million Comcast Digital Voice customers.

Comcast Developer Network

Matthew’s latest project is the Comcast Developers Network a Comcast-scale secure web services platform for the development of cool new media and entertainment offerings. The Comcast Web Platform environment generates of billions of software events each day from caching and load-balancing, origin application servers, databases, middleware and content delivery networks for images and video streams. Comcast services demand high quality. Much of the Comcast content is exclusive and premium services drive revenue. Interfaces between technology components (applications, delivery platforms) need to adhere to best practices to ensure the highest degree of end customer experience.

Why Splunk?

Comcast has acquired many system and application management platforms over the years, but nothing was providing the team with the robust information from operational telemetry the teams around the company need to ensure data integrity, stability, application quality and efficiency. Several efforts specifically drove Comcast to consider and deploy Splunk.

• Product rollout: The team wanted the ability to predict and correct potential issues before going live into into production—Splunk has become a required best practice for new product rollouts.
• Network/ System Integrity: Understanding security and user experience across a very large network and set of systems is a must to protect the business. Splunk provides the insight the network and system teams need across many different silos of technologies.
• Business Intelligence: Having immediate access to real-time events and historical trends allows the various Comcast business teams to react quickly and adapt to changing customer behaviors.
• Agility: Alerts and Dashboards indicate discrepancies so distributed teams can investigate immediately and remediate failures and attacks.

Video CDN/CMS Performance

“In content management systems and delivery networks a devil walks the long tail. If you’re facing concurrent hits across the tail of the curve, sharpen your pencil, you’ve got problems!”

Splunk helps Comcast understand the risks of instability in our systems, especially during periods of high concurrency. Through pre-production modeling of even patterns and subsequent monitoring of these patterns Splunk pays for itself by helping Comcast avoid deployment of vulnerable systems, downtime, and upset customers.

Predicting System Imbalance

Comcast has successfully used Splunk to evaluate potential infrastructure vendor’s solutions and determine if they will balance loads properly across a large, indeterminate infrastructure. Often the answer is no as illustrated here in a Splunk report of resource utilization across various services.

Splunk has also been utilized to see whether solutions will be resilient to different traffic patterns, helping the company perform predictive analysis before making critical infrastructure investments.

Load testing is performed during non peak hours and the results are analyzed for system failures over time using the telemetry data Splunk can correlated across various logs, messages and events.

When failures are found the Comcast team uses Splunk reports to dig deeper into the data.

Security and Compliance

In addition to operations use cases, Comcast security and compliance teams leverage the consolidated logs across data centers to enable faster threat assessment and security monitoring.

• Monitoring for bad actors to trigger alerts,
• Conducting threat detection over time,
• Detecting attacks/vulnerabilities in systems and
• Auditing systems in support of security assessments and compliance.

What’s Next?

Next up for Matthew and team is the launch of the Comcast CodeBig Platform enabling a network of developers to create content for the network. Some of these developers are already using Splunk in their own managed services like Mashery. Comcast is working to hook the Mashery Splunk installation to their own in-order to provide visibility across multiple services and providers of content and entertainment functionality.

Chris Abboud manages the Enterprise Systems Management team at Dow Jones — monitoring customer facing infrastructure and applications. Dow Jones provides global business news and information services to millions of consumers and enterprise media groups. Keeping these revenue generating services running 7×24x365 is the highest priority. Chris also manages the DJ service management platforms (Remedy, Knowledge Base, etc.) He’s been with the DJ organization for 10 years, in current role for 3 years.

“Our mission is to address issues before they become service impacting events. Failures are going to happen — we need to make sure people know about them as soon as possible.”

The Splunk Set-up

The Dow Jones Splunk installation includes

• Data from 6000+ servers globally,
• 13,500 + source types,
• 1,700 network devices (primarily Cisco and Juniper) and
• Ten distributed Splunk servers in difference geographies index ~100GB a day and provide a new global logging console.

Why Splunk?

Each Dow Jones command center now has the ability to know what’s happening before customers do across a wide range of internal and external services. Splunk speeds the time to resolution for email outages that may impact internal users’ productivity and editorial sites downtime that can directly impact to customer service and revenue. Dow Jones has found Splunk generates significantly fewer false positives than traditional monitoring systems and new resources are much easier to manage and deploy.

I visited Sydney and Melbourne last week to host our first Splunk Live events in Australia. Its my first visit to Australia and I’m really blown away by the friendliness of the people we’ve met. And the “Australian for Grep” t-shirt finally had a proper home. Attendees at today’s event in Melbourne and Tuesday’s event in Sydney included an impressive list of current customers and partners and a number of new users evaluating Splunk for the first time including Telstra, Ericsson, InfoSys, Frontline Systems, Fujitsu, GE Capital Finance, Toll Holdings, Vanguard Investments and more. We owe a huge thanks to the team from Digital Networks Australia who sponsored the two events.

Martin Brown, A Large Australian Financial Services Company

In Sydney Martin Brown, pictured below with me, gave an excellent presentation on using Splunk for Identity Management Compliance. Martin is a Technical Architect managing the development and operations of the world wide web application security system‏ for a major financial institution. He’s had many career evolutions from implantable device electronics and software engineering, UNIX and network systems administration, internet systems management and security.

Martin’s company has a requirement for presenting client security history from their web applications and to be able to access this information to look for suspect IDs from the past six months. Tivoli Access Manager (TAM) is used for both external and internal identity management and access control. More than 200,000 clients authenticate externally through TAM.

His Splunk deployment is very much out of the box with a range of saved searches and some role partitioning. It consists of a single Splunk server with 1TByte of local disk for retention. The TAM logs are rsynced regularly and directly mounted from various hosts and systems. 12 internal and 12 external TAM hosts generate 5 GB/day of data or ~2TB of data a year.

The current user base consists of business second level support teams and TAM support group for third level support. The user bases is expected to extend to the Risk Management Group and first level help desk support soon. Their classic use case is

“Client X’s account has been compromised. What applications has he/she logged in to in the past 6 months?”

The old way required days / weeks of work and support from multiple teams. Often needed to pull in log files from offsite backup tapes then grep through GBytes of data from several hosts. Fun fun. Now with Splunk Martin’s team finds answers in minutes and soon will train Tier 1 agents to do the same, eliminating the hassle of Martin’s team fetching data for everyone. Next he plans to add App server, Web Server and Load Balancer data, role partitioning to restrict business user access to relevant logs, off-shore implementations to present local application logs, API consumption for helpdesk one-stop-shop interface.

Nick Clark, Ericsson

Nick Clark is a Technology Manager in the Solution Management & Utilities Consulting, System Integration & Multimedia practice with Ericsson where the focus is on bespoke support and life cycle management services for complex infrastructures. His group focuses on mobile and fixed network infrastructure, telecom services, software, broadband and multimedia solutions for operators, enterprises and the media industry. He presented his Splunk solution which Ericsson implemented at Telstra in the mobile multimedia services area to troubleshoot problems and investigate incidents. The solution was initially implemented to provide coverage of the 2008 Beijhing Olympics. Telstra predicted massive interest for mobile streaming yet demand exceeded all expectations. Splunk helped Ericsson and Telstra quickly pinpoint, manage and address problems. Because application failures and limits were discovered before they cause serious downtime Telstra maintained an uptime above 99.9% during the Olympic Games.

Telstra manages more than 10M users and 50 plus content providers on the Telstra Service Delivery Platform providing multiple mobile portals, content transformation, mobile streaming services and device specific rendering and UI over 2G and 3G networks. The environment consists of 60+ servers (Solaris 9/10, Windows 2003) and many platforms and technologies providing service orchestration, rich media content management, encoding and streaming for terabytes of active content.

Ericcson and Telstra’s challenges before Splunk were numerous including:

• no central view of logs and events resulting in difficult to troubleshoot problems,
• support and operations diverted to log fetching and ad-hoc reporting delaying work on high priority projects,
• no consistent approach to log handling and storage making it difficult to locate, access and archive logs and
• poor visibility of service and transaction flows extending outages.

The Ericsson team chose Splunk to help Telstra gain a holistic view of the environment, troubleshoot outages more quickly, provide users with ad-hoc reporting and control access to logs with by role. They are currently indexing roughly 20GB per day on a dual processor, dual core Xeon GHz server with 16GB of RAM. 30 support people (tier 1 and up) currently Splunk application, server and network logs and events to troubleshoot problems. The team makes extensive use of Splunk tagging to create alerts for future notification of problems reoccurring. Perhaps the most valuable thing Ericsson has done with Splunk is track end to end transactions on the Service Delivery Platform. With one view across all services and transactions to track activities the team can finally provide transaction level alerting and reporting.

Thank you again to Nick and Martin for presenting so well and Monsour, Martin and Sky with DNA who did a fantastic job and are representing Splunk very well down under.

I’m finally getting my head above water after a tireless run up to and hectic week launching Splunk 4. The highlight of the launch for me was Splunk Live London. IMHO Splunk Live London 2009 was unrivaled as the most outstanding Splunk event yet. We came up with this idea of getting local customers together as a way to launch Splunk 2 in June 2007. Five of us Splunkers sprinted between eight different cities in two weeks to share what was new and encourage users to exchange stories of how searching their data centers was changing life for the better. Its an exhausting way to launch a new product, but it worked so well we’ve integrated Splunk Live events into the mainstream way we do business and interact with our community. I’ve long since lost count of the number of Splunk Lives we’ve conducted all over the world including places like Cape Town, Johannesburg, Beijing, Tokyo, Singapore, Bangkok, Sao Paulo and yes once again in London.

This year’s London Splunk Live was really special. The event occurred during our launch of Splunk 4 and surpassed our expectations as the largest event we’ve ever held. More than 100 customers and users attended at the Cumberland Hotel and their swank conference facility, complete with a business canteen like breakfast experience, near Marble Arch in West London.

But the dominant reason to attend any Splunk Live are the presentations and round tables with forward thinking IT professionals who are using Splunk to transform the way they manage IT. This year we were very fortunate to have three Splunk customers who took time out of their busy schedules to come to London and share their experiences with us.

Accenture - Alexander Strobl, Technical Consultant

Alexander has been a visionary inside Accenture bringing the power of IT Search to enterprise clients in Germany where he works for Accenture as a Technical Consultant in the Data Center Technology and Opeations team. Alexander is responsible for analysis, design, roll out of Splunk. His most recent Splunk project was with a large worldwide services company with more than 50,000 employees on three continents operating mail order, distribution, e-commerce and over-the-counter-retail trade. Accenture implemented Splunk to transform the management of several technologies including Linux, virtualization and large-scale storage systems.

The project was part of an IT project to reduce the time to triage problems and improve quality of service. Challenges were:

• no centralized access to logs and events,
• critical IT data was stored on local file systems which were copied to central storage only once a day,
• manual processes to locate errors,
• no correlation between events on different services/servers and
• development time was spend building workarounds rather than working on revenue generating applications.

All of this resulted in complex and time consuming analysis and end the end long MTTR.

The Accenture Splunk installation is currently indexing ~50GB/day including custom application files and events from 10+ integrated business critical applications and services. There are two Splunk indexes; one for testing and one for production environments and the team has established interfaces between Splunk and several other legacy data center tools.

Telenor - Henrik Strøm, Security Architect

Telenor is Norway’s largest ISP, Mobile Operator and Telco. Its one of the largest mobile operators in the world, with 160+ million customers and was founded in 1855 - 154 years ago. The company has 13.000 employees in Norway and 26.000 abroad. Telenor has been rolling Splunk out for centralized log collection and management using Syslog to forward data where it is already in place and using Splunk as a forwarder for new systems and systems with complex multi-line and/or XML structures Syslog can’t handle. Sources of data handles by Splunk include:

• application logs (Web, Email, IPTV)
• data center logs (server, network, storage and firewall)
• IP backbone logs

Use cases include what Henrik refers to as digging, dashboards baselines, alerting and reporting. One of the best “digging” examples Henrik mentioned was identifying Unix Kernel Errors over the last 30 days. This kind of information routinely went unnoticed prior to Splunk’s arrival.

Another powerful use case explained by Henrik was how to baseline what is normal in your environment. For example, how many errors do you have on average for a particular type of device (routers, servers, specific applications, etc). Splunk was used to baseline normal Linux kernel behavior and found roughly 20 kernel errors per Linux running instance every 15 minutes.

The base line then allows the team to schedule simple searches to look for deviation from the baseline and send out alerts before downtime occurs from these hidden sways in behavior. In one case Splunk found thousands of errors occurring on a specific type of device, where the normal baseline was around 20!

The Telenor team also uses Splunk to identify and report on security situations that may impact their customer facing network and services. Because they are able to easily compose dashboards showing for example which Web servers are under attack and who is attacking them all in one place, the team saves Telenor from potential downtime, performance degradation or theft of data due to attacks they’ve not seen before and are missed by existing security policies and technologies.

Vodafone - Paulo de Carvalho, Network Services Manager

Paulo de Carvalho has been using Splunk at Vodafone for almost two years now. His presentation titled “Freeing Information from Organizational Silos” lifted the idea of leveraging logs and IT data out of the realm of just system administration into a thirst for higher level intelligence that crosses not only IT but also business functions. Paulo started by describing the current service oriented architecture (SOA) at Vodafone and how attempts to objectize and re-use capabilities creates incredible complexity among the services, technologies, processes, tools and people.

The past couple of days I’ve been visiting China meeting with some of our technology and channel partners. It just so happens I was present in Beijing for the 20th anniversary of the 1989 Tiananmen Square Events. Yes it really did happen despite what the Chinese government says. Speaking on Saturday at the F5 APAC Sales Kickoff I found myself staying over the weekend with Sunday off to roam around Beijing like a tourist, something I rarely get a chance to do on business trips. It is amazing to me to see how the Chinese and Taiwanese work on Saturdays. In the US we rarely see that. Europeans chastise Americans for working too hard but I guess they should really see the work ethic in Asia and then we’d look more normal.

Watching the 2008 Beijing Olympics last summer things there certainly seemed more normal than 20 years ago, but being there in person with all the festivities gone things seemed really strange to me. It is very difficult to describe. Maybe I was jaded by all the newspapers I’d read on the way to Beijing. On a nice long 13 hour flight from Washington DC with plenty of reading material I consumed James Kynge’s piece in the Financial Times questioning whether the Western media really understood why the student demonstrators were protesting. He went on ascribing the word “democracy” with the student motivations and questioning whether we or they really knew what it meant despite the fact that he spells out their desires in plan old English which sounds like democracy to me.

“Almost everything fell within its scope: campaigns against corruption, nepotism, inflation, police brutality, bureaucracy, official privilege, media censorship, human rights abuses, cramped student dormitories and the smothering of democratic urges. But to say the demonstrations were to “demand democracy” is an oversimplification.”
James Kynge, Financial Times

It’s almost impossible to describe the strange feeling I got while walking through Tiananmen Square observing the soldiers and the huge portrait of General Mao that dominates the landscape. Maybe part of it was due to the increased tension of the anniversary. Maybe not. Tiananmen has come to symbolize the unspoken and largely unrecognized tension between the economic progress driving modern China and the old fashion communist government still ruling there. The Chinese seem to have a foot in both camps. The eeriness I felt came not only from my surroundings and an understanding of the principles they stood for but also from the reaction of my Chinese and Taiwanese friends. Their usually jubilant outgoing personalities were completely subdued in the square. Was a sign of respect and mourning that drove their thoughts? Perhaps to some extent. But in quiet whispers and conversations out of the ear shot of any “green” uniformed soldiers (versus the “blue uniformed” security guards they confessed to being actually scared to speak for fear of someone or something listening. Challenging them I said, “surely you must be joking.” But it was no joke. Only when we crossed the street into the forbidden city did their usual personalities return.

Of course this began a prolonged conversation over the next 24 hours as we visited the great wall, a new Beijing restaurant and departed through the impressive new Beijing airport. I kept asking and trying to understand. How can a country of so many people be controlled by the minds of so few? What are the real limitations to speak out? And what effect will economic progress have on the political future of China? There was no shortage of stories supporting the fact that the government still does take a very heavy hand to those who disagree. But rather than discuss it, everyday Beijing seems to sweep the event of 20 years ago under the rug. As one of my Chinese friends said, “everyone is embarrassed and we just pretend it never happened.”

At the same time I was traveling through out China, the articles started pouring in about Beijing’s efforts to step up Internet and IT censorship. Upon reading the perspectives pouring in about “Green Dam” I was reminded of the impact the technology industry is having on the whole situation. It was bad enough I couldn’t get to sites like Twitter and Youtube form my hotel room. Now the Chinese government is requiring every PC sold in the country starting July 1st has to have special software blocking all sorts of things. The move is being presented as an attempt to protect children from online pornography but is obviously one more attempt by Beijing take its censorship to a new level. China currently has the world’s most sophisticated and multi-layered system of Internet censorship. Objectionable content on domestic Web sites is deleted or prevented from being published, and access to a large number of overseas Web sites is blocked or “filtered.” Decisions about what to censor are based on the Chinese government’s attempts to control the minds of 1.2B Chinese. There is no transparency or accountability, no public consultation in developing block lists or censorship criteria, and no way to appeal the blockage or removal of Web content.

In a notice to PC makers, the Ministry of Industry and Information Technology said all PCs shipped in China needed to offer Green Dam/Youth Escort, identified as a “green internet filtering software”, either pre-installed or as part of basic software packages. In May 2008, the government picked Jinhui Technology and Dazheng Language Technology, two Chinese software companies to develop the software, according to a contract award notice from the MIIT. While these companies claim their software is only being used to block sites although last year, researchers discovered that a Chinese version of Skype contained the ability to block politically sensitive words in instant messaging chats, and to keep a record of the use of such words.

Things are moving pretty fast at Splunk and I wanted to comment on the exciting news we announced last week.

In 2004, myself, Erik Swan and Rob Das started Splunk with a vision to battle IT complexity by embracing it. We were thinking of things a bit differently. A different way to address the management of IT by applying search to millions of data center artifacts. Traditionally these artifacts were summarized, filtered and reduced and then forgotten - leaving us humans in a pickle when we needed to figure out what’s really going on. For us Splunk was also about a different way to interact with the market taking an approach of utter transparency. Our public product road maps, freely downloadable software and straightforward marketing had even our early stage venture capital investors thinking we were crazy.

By start-up standards, we seem to have succeeded. Splunk now has more than 250,000 user downloads, more than 750 enterprises, service providers and government agencies worldwide as paying customers and a growing list of partners who embed Splunk into their software, hardware and managed services including companies like Cisco and British Telecom. According to my venture capital friends, very few start-ups make it to where we are today. But, fueled by a love for innovation and so many passionate users we’ve challenged ourselves to see beyond achieving success as a start-up. We believe Splunk can be a company that gets the IT industry thinking differently.

Creating change isn’t easy and we’ll need all the help we can get. Fortunately, we’ve been blessed with an ability to attract top talent at all levels. But our most recent success tops them all. Godfrey Sullivan has joined us as our new President and CEO. When you meet him you’ll realize the incredible passion he has for building great companies. Most recently he was President and CEO of Hyperion Solutions. He took Hyperion over a period of six years to $1B in revenues. Hyperion was acquired by Oracle in 2007 for $3.3B. Godfrey also serves on the board of directors of Citrix Systems, Inc., and Informatica Corporation. Just as important as his business and leadership abilities, Godfrey has the cultural DNA that fits right in at Splunk.

Here’s the yin and yang that is Godfrey. He owns one of only 4,038 1994-1997 Ford GTs. Now this thing is fast, really fast. 0–60 mph (0–96 km/h): 3.3 seconds 0–100 mph (0–160 km/h): 7.3 seconds Standing 1/4 mile: 11.2 seconds @ 134.2 mph Top speed: 212 [11] And his other car is a Toyota Prius. Enough said. Godfrey couldn’t join us at a better time. We’re scaling all aspects of the business and need the leadership of someone who’s been through this type of explosive growth before. For me personally, it’s pretty cool to work beside someone of his experience, talent and steady as she goes outlook on life. And I get to continue to do what I do - build things. I’m now leading the team building our partner ecosystem working with Developers, MSPs, Resellers, Technology Partners and System Integrators around the world.

Of course this hyper growth wouldn’t be possible without your passion and support. Thank you all for that.

Happy Splunking!

This week we’ve been moseying through the Southwestern part of the US with our Splunk Live show. We changed up the format a bit with Splunk technical workshops in the morning and customer round tables in the afternoon. The technical workshops were a big hit with more than 200 people registered to engage with our Splunk Experts. During the workshop you were able to download, install, configure and start using Splunk on your laptop or server with remote access. The best part about Splunk Live events though is sharing ideas with other Splunk fanatics.

Ryan Peterson from Infusionsoft, a marketing automation company, gave a great talk in Scottsdale about his Splunk deployment for the company’s email infrastructure. Ryan is tasked with keeping more than 12M emails a week flowing out of the system to support Infusionsoft’s Automated Follow-up Technology (AFT). Ryan has multiple servers in different geographies in addition to PCI Compliance requirements. He demonstrated using Splunk to troubleshoot problems spread across the messaging infrastructure, address reporting inaccuracies and deliver PCI reports to auditors. He’s even indexing the content of email with Splunk using a scripted LDAP data input. Cool stuff.

In San Diego Tony Doan of the Genomics Institute at the Novartis Research Foundation (GNF) and Eric Van Johnson from Sony Consumer Electronics joined us. Tony is a security engineer and former pen tester. He also confesses to be a recovering Unix sysadmin. GNF has 600 Windows desktops and several hundred Windows and Linux servers supporting the discovery of new biological processes and improved human therapeutics. Tony discussed how they splunk Cisco CSC, Bluecoat, Symantec AV, Arpwatch, Cisco Switches and Wifi access points to find what he calls “previously unknowns” to improve operational availability and security. He says they’re finding new uses everyday but Tony’s favorite is splunking Cisco IPS and Cisco MARS events looking for odd behaviors. Next up for GNF is eating Windows Event Logs and Windows Registry inputs together with summary indexing for consolidated reporting.

Eric Van Johnson is the eServices Hosting and Operations Manager at Sony Consumer electronics. He led an great discussion on splunking IBM Websphere and MQ Series events including how Sony has integrated operations and development environments to identify problems with complex apps more quickly and avoid unnecessary escalations to the development team. He shared with us Sony’s roll out of Splunk to their Business Intelligence Group. The idea is to complement aggregated WebMethods data reporting for business activity monitoring. Next up he wants to feed Splunk data back and forth with Verizon’s hosting operations since some of the Sony servers are hosted at Verizon and Verizon is also using Splunk.

In LA Rich Horace, Director of Systems Engineering and Operations at Fox Interactive Media demonstrated how Fox uses Splunk in the Fox Audience Network. Basically these are the guys that serve web advertisements across all the Fox properties including MySpace, Rotten Tomatoes, Fox Sports and IGN. He’s challenged with launching new monetization platforms and keeping the existing ones running. Rich gave a fantastic overview of his Splunk installation which consolidates/aggregates data form disparate systems in order to protect against hackers and meet PCI and SOX requirements. He currently runs an environment with ~600 Linux servers, load balancers, servers, NetApps and network switches. So far he’s indexed 1.5B events. We engaged with everyone in a lively discussion about securing production sites from developers and controlling and auditing access to data using Splunk’s access controls and search filters. Rich also discussed how Fox is using Splunk to integrate with various Citrix products including Netscaler and XenApp.

Thanks to everyone who shared their stories with us this week, it was really awesome.

It’s Sunday night before the start of our first ever Splunk Developer Camp. Never before have we invited developers from our community at large to participate in sharing their ideas about building Splunk Apps and learning about all the cool stuff in our upcoming releases. I think I can speak for everyone at Splunk when I say we are truly amazed with the level of interest and participation. We’ve had to move the venue three times now to accommodate the growing list of participants and while we initially expected the mix would be mostly existing customers, we’re really pleased with the mix of developers coming tomorrow. 125 Developers 91 Organizations 26 Industries 9 Countries Only a third of the developers showing up are customers. The rest are system integrators, MSPs, OEMs, ISVs and VARs.

Post Camp Update

We’ve organized the day into a combination of an un-conference format with developer round tables, sneak peaks of future versions of Splunk, demos, demos, demos from customers and partners and training on the Splunk API and SDKs. Our goal for the day was to both educate campers on how to effectively build Splunk apps and to get everyone jacked up about the possibilities. We broadcast the sessions live on Splunk TV.

The day started with a quick intro by me. I gave everyone a brief Splunk history lesson of the past five years and demos of the Splunk for PCI and Splunk for Server Virtualization applications. I wrapped with a discussion of our strategy to seed Splunk everywhere and to enable developers to distribute their applications to Splunk installations around the world in the near future. More on this in a future post.

Erik Swan and Rob Das, my two co-founders followed with a more in-depth evolution of Splunk chat which many focused on all the weird prototypes and company names we thought of before the real Splunk. Some of it is funny and some down right scary. Amazing what guys out of a job can come up with.

Konfabulator Follow Along

Next up Kord Campbell, Director of our Developer Program gave an overview of agenda for the day and reviewed how to register with the Konfabulator and follow along with the many demos up on our SplunkLabs EC2 server at Amazon Web Services. This worked great as everyone could build and run the demos on their own EC2 instance. Kord also showed off the new Splunk Wiki for developers and application users. We’re in the process of moving all our documentation to the wiki as a one stop shop for information on using, administering, deploying and developing for Splunk. A few other Kord matters included the review of our new Developer Program additions including a 2GB Developer Enterprise License for registered developers.

Splunk Apps

Jef Bekes, our Head Designer and Raffy Marty our Application Product Manager then gave a very inspiring talk about the future of Splunk and Splunk Apps. The basic point being in Splunk 3.3 today there is no sense of application context. This means the same default user-interface for all applications and that all knowledge (saved searches, alerts, reports etc.) is shared across all installed apps. It’s impossible also to “switch” from one app to another. Splunk 4.0 attempts to address this whole problem by making applications first class objects that can be containers for collections of other objects at the interface, knowledge and configuration layers. As more an more Splunk applications arrive on the scene this encapsulation becomes increasingly important. Jef and Raffy showed a sample Splunk 4.0 Help Desk application that included custom branding, restricted task-based navigation and structured search user interfaces and results views. Other Splunk 4.0 features were reviewed too; Splunk Web gadgets, the Application builder, improved charting and content grouping.

Developer Platform and API

The Splunk Developer Platform futures was up next with Tom Donahoe, Splunk Product Manager and Johnvey Hwang Lead UI Developer. Topics included the Splunk 4.0 improvements like Application Builder, REST API Additions, UI Extensibility and SDK Support. The Application Builder eases application creation and packaging dramatically improving the experience beyond where Splunk 3.3 currently stands. The Application Builder will be available in both command-line and GUI to provides application configuration isolation and leverage file system security controls. Johnvey reviewed with us planned REST API additions for 4.0 like

• Alerting: history, status, improved generation
• Notifications: email, RSS
• Search scheduling management
• Knowledge management
• Authentication: users, roles, single sign-on
• Distributed: topology data, server metrics

Splunk Ninja

The Splunk Ninja (aka Michael Wilde) graced us with a visit and showed off his demo Godness with a Zero-to-Lightspeed set-up and data eating with the new Splunk Crawl feature in 3.3. Sweet!

Search Language

David Carasso, a Senior Developer and Alex Raitz one of our Solution Architects did a fantastic overview of the Splunk search language and ran through some really cool examples of powerful stuff like

• What’s the most important hard disk error on each of my hosts?
• Who sent me the most email?
• How long do users stay on my website?

David showed us how to create our own search commands too. Awesome stuff.

Large Scale Reporting and Summary Indexing

Steven Sorkin, Head Indexing Geek led a wonderful talk on large scale reporting using great examples like finding violations in security data on application layer firewalls and routers. He covered how we use map/reduce models to summarize batches of events - what we call summary indexing. It turns Splunk into a sort-a time slinky.

REST/ATOM API and Splunk Gadgets

Recently Matt Asay wrote a thoughtful piece about how some technology companies are consumerizing the computing experience. In the case of Apple, Business Week writer Peter Burrows has also recently wrote about The Mac in the Gray Flannel Suite exploring how CIOs are testing the appetite for Macs in the enterprise. Michele Goins CIO at Juniper Networks recently ran a test among the company’s 6,000 employees discovering that 25% wanted a Mac.

Consumerism of the enterprise computing experience is well underway with Apple, Google, SalesForce and even Cisco’s TelePresence and WebEx offerings. According to Matt, all of these products delight users with a positive user experience by focusing on adoption first and dollars second. “Simple, fast and useful,” is the key.

Could it be that the consumerization of IT is far behind? How many enterprise management vendors focus on adoption first and dollars second? Can you honestly say that any of your vendors put you and your users first? Do the words “simple, fast and useful” come to mind as you’re writing the check for your maintenance renewal every year?

We recently compiled the feedback from out Q1 customer survey. Each quarter we survey our customer base like most companies do. What’s perhaps different in our case is we focus intensely in our surveys on the user experience with our product. We ask about ease of use, administration, upgrade processes and documentation quality. What we continue to find is users and customers actually like using Splunk versus being compelled to use it by their organization.

Maybe we’re participating in the consumerization of IT. Perhaps we just like using the stuff we build. Regardless, we are constantly working to improve the Splunk user and administration experience. To us this is the #1 measurement of our and our customer’s satisfaction. You may already know we post our product roadmap on our website including where we’re focused for the next several months. If you have your own ideas send us your feature and improvement suggestions directly to Splunk support.

The US economy is heading into a recession and technology spending is in for a steep decline in 2008. So every major prognosticator and news outlet from the Wall Street Journal to the Financial Times would have us believe.

Are these people watching the same movie I am? There are two problems I have with this economic hyperbole. Yes that’s what it is. I guess it sells newspapers and gets people to watch things like CNBC. But boy is it misleading.

First of all, in macroeconomics, a recession is a decline in any country’s gross domestic product (GDP), or negative real economic growth, for two or more successive quarters of a year. Yet nobody that I’ve read is forecasting negative growth. They’re forecasting a potential slow down in growth from the current 3.5% per quarter to 1.5 to 2.5% per quarter. But the news outlets feel compelled to use the “R” word just to get attention. Totally irresponsible.

On to my second gripe. With regards to technology and IT spending, I believe, based on what I see, we are in beginning of a long-term gradual increase in IT spending within large enterprises that started eighteen to twenty four months ago.

Sure the current credit crisis may have a short-term impact on budgets within Financial Services companies, but I don’t see any slow down yet. The major consumer, commercial and investment banks we work with have so many critical, revenue generating IT projects in backlog I fail to see how spending is going to slow at all. The telecommunication sector is finally back on the mend after the post early 2000’s bubble and hangover.

Social media, online shopping and the always on dimension of the Internet have online services and large Internet sites like MySpace and Amazon accelerating software, hardware and services spending just to keep up. And security, privacy and compliance initiatives and mandates have companies, service providers and government agencies increasing spending on these items by some 20% or more in 2008 to try and limit their exposure and risk.

Just a month ago the Financial Times had a great piece entitled “What’s on CIO wishlists?” Here’s a quick summary.

1. Business alignment and strategy
2. Hiring and retaining the best staff
3. IT innovation/new methodologies
4. Security
5. Collaboration technologies
6. Controlling costs
7. Compliance and regulation
8. Virtualisation
9. Customer service
10. Mobility (Green issues came 11th)

Doesn’t look like a slow down to me.

I’ve written previously about our experience this year raising a $25M Series C round of venture financing. Venture Diaries: Part One discusses why you want to think before you act and investigate who to target as potential investor partners. Venture Diaries: Part Two looks at how to perform your investigation. In this third part, I look at how to handle the horse race that inevitably develops once you get a few term sheets.

For me it all started when the first term sheet came in. Funny how some VCs still use fax machines. I had to go figure out where ours was. In the current seller’s environment (yes that’s what you are, a seller of equity in your company) one thing to keep in mind is your first term sheet will just be a starting point. Expect that it will probably be lower (perhaps significantly lower) than where you want to end up. Also expect once the first term sheet comes in things will really start to heat up. Nobody wants to miss out on a good investment and VCs are just egotistical enough to really help your cause. However, you should realize each VC has their own style. Some will try to move first in hopes of stealing the deal from others. Others will try to wait till the end and trump any offer — figuring the last hand in has the best chance.

This is where the entrepreneur’s job gets difficult. You want to put everyone on notice that you have a term sheet. This way things really get moving and you can quickly figure out who is really interested and who is just playing along. But what process should you use? How do you maintain your integrity when everyone is asking you for information.

The analogy of selling a home comes to mind. Some sellers will run a sealed bid process. “All offers are due on Tuesday by 5pm and the top offer wins.” This tends to work better in real estate because you already have an asking price. Buyers know what minimum price you expect. In addition, most markets have an established bid/ask ratio where homes get sold (unless your in a rapidly declining or accelerating market which isn’t often the case).

When you’re selling equity in your company to venture capitalists the number one rule is don’t, under and circumstances signal an asking price.

You will get hammered by investors wanting to know what your expectation is for your company’s valuation. There is one and only one correct way to answer this question every time. “We believe we’ve made significant progress since the last round, but the market will price the deal.” This way you signal you’re expecting a nice increase over the last round price but you don’t set a ceiling on this round’s price. Trust me they will all ask you over and over and over again, but don’t give in!

Back to process. Sealed bidding doesn’t work. So what does? I call it the Road Runner strategy. Remember how the Road Runner used to always chase Wile E. Coyote to the edge of the cliff and then watch him fall off?

This is what you need to do with each of your potential investors. To maximize your terms and perhaps most importantly figure out what it will be like to work with each of the potential VCs you have to push them to the edge of their comfort zone. While sometimes uncomfortable the process will show you what your potential new board member and investor is really like. Chances are the way they handle a competitive negotiation is the same way they’ll handle themselves in difficult board meetings.

Start out by telegraphing the fact that you have a term sheet to the other investors looking at your company. Be careful not to disclose any of the terms, but tell them it is a competitive offer. If the terms are clean, telegraph that as well. In my case I found it helpful at this point to set a deadline a week or two out whereby everyone must wrap up their due diligence and get you a term sheet. It’s actually a good idea to have a soft deadline communicated in your first meeting with each investor. This way nobody is surprised when you reinforce the deadline. You’re deadline will be soft, but make it seem firm without being pushy.

This is the point where you need to be in constant communication with each interested investor. Return phone calls and emails within an hour. Make sure everyone knows you are available to get them any information they need.

Chances are the VCs will really start selling you at this point. Remember all those tricks Wile E. Coyote had? Most of them some type of Rube Goldberg device manufactured by Acme Corporation. Like the Coyote’s tricks, most of the VC’s points about why they’re the best are somewhat fictitious and sometimes totally outlandish. But none the less they’ll try. You’ll hear all sorts of stories about why you should take a lower offer and how each investor needs to own a certain portion of your company in order to dedicate the time to sitting on your board. Listen attentively, thank them all and then remind them of the deadline and ask them to make their best offer.

According the National Venture Capital Association (NVCA), there are 798 venture capital firms managing more than $235B in the United States. These are long-term, professional investors who specialize in funding and building new, innovative companies.

So how do you figure out who to approach for funding? This is the area where I find entrepreneurs make the biggest mistakes. Most of us approach investors we know. Perhaps you have a friend who knows a VC or you have a friend who is a VC. How do you know if your friend or the person you get introduced to is the right investor for you? Most likely they’re not. Not all VCs are alike. Some are geared for early stage and some are not. Some are suited for late stage investments while others just say they are.

You can’t always trust what an investor says their appetite is either. I’ve pitched to investors who say, “yeah we do Series A” only to be barraged by questions like, “how many paying customers do you have that we can talk to.” On the other hand, I’ve presented to wanna be later stage investors that were only prepared to pay an early stage price.

You need to do your own research. Venture capitalists are for the most part, creatures of habit. They don’t change investment philosophies much. Often within a firm it will take a generation before new blood arrives and can affect major change. In addition to the succession challenges, VCs are bound by the structure and economics of their business. Venture funds are seven to ten year financial vehicles. VCs raise the money for their funds based on an investment strategy which takes several years to play out.

I suggest doing your own primary research. Identify eight to ten prospects with a track record of backing entrepreneurs like you. Look for a history of focusing on your market and the stage your company is at and the type of involvement you want. Suspend your judgment during the your data gathering. Just get the data and avoid acting surprised or judgmental. Get specific data on the number of projects and stages of investment each firm has completed recently.

When we raised a Series C round earlier this year, I identified eight firms to approach based on their past investment history. Specifically, I was looking for firms and partners that had done a majority of their investments in late stage, infrastructure software companies over the past eighteen months. I wanted to focus on VCs who demonstrated a track record of paying a fair price to invest in revenue generating companies that need capital to accelerate growth. I gathered data on how many investments each VC made, how many of the investments were later stage and how many later stage investments they actually led versus just participated in. My goal was to focus on investors with the highest percentage of later stage deals led as a function of total investments made.

Of the VCs I researched the percentage of Series C or later deals led ranged from 15% to 95% of the total deals invested in during the prior 18 month period. Surprisingly the firm with the 15% invested in far more deals and far more later stage deals than anyone else. But the participation in later stage deals was mostly follow on investments in their existing portfolio. This was not the type of later stage investor I was looking for to lead our financing.

There were two VCs that approached us and pitched themselves as later stage investors. But the data just didn’t support their claims. The one had a 19% rating and the other a 17% rating. Despite showing great interest both of these investors dropped out of the financing process when we had several term sheets and commented, “the price is too high for us, we can’t dedicate our time to the project unless we can own more of the company.” At which point the leopard really showed his stripes.

The core set of later stage VCs I focused on had ratings ranging from 50% to 95% indicating they had led a significant number of later stage investments in the past 18 months. Every one of these investors delivered us a term sheet at a competitive price.

How do you find this information? The brute force way is to visit a number of firm’s websites and go through their portfolios. This takes a while but can yield the information you’re looking for if you put in the time. It is certainly a lot less time consuming (and less humiliating) than pitching investors that will never invest in your profile situation. There are a variety of venture capital databases that can make your research much faster and easier. If you have a friend that’s a VC they likely have access to one or more of these sources. If the answers about a particular firm are vague drill down and get the real story. If you can’t figure it out, move on. You’ve got 798 firms to choose from.

I’m not sure if it’s the start of a new quarter, the full moon or my two seven year old boys that have me thinking about this, but we seem to be blowing a lot of things up lately. A few examples…

1. We blew up our product development process
2. We blew up lots of our software
3. We blew up our business planning process

When I say we “blew ________ up” (enter your own thing here) I mean we decided to take another course of action, look in the other direction, put other people in charge or just plain start over from scratch. Combustibles are exciting for lots of reasons (especially to second graders) but as a new type of business tool?

I’ve written in previous posts about our move to an Agile product development process. This required us to literally discharge our old way of taking input from customers, scoping features, planning releases and testing. Of course it also meant we had to ignite our underlying work flow and tools supporting product development. It all made me a tad nervous : { For more than a month I couldn’t tell you what would appear in our next release or when the release might be available for download. If you use Splunk, you know that we live and die by our product road map and release schedule. During that month our engineering, qa and product management teams went through a metamorphoses. They moved from being top down, planning driven to bottom up, innovation driven. We had reached the point where we couldn’t plan or prioritize features. The old process of having a team set out a plan and working towards a release wasn’t working anymore. So we blew it up. Now we have a process where by parallel scrum teams work on various facets of the product and they do the planning, constantly. It’s interesting how nobody, but yet everybody is in charge. The initial results are just in. Splunk 3.1 will soon be available for download in a mere eight weeks after Splunk 3.0 was posted. And Splunk 3.2 will be released in beta eight weeks from now. That may not sound like much but when you look at the amount of innovation in each release, the speed with which we’re moving enhancement requests from the field into features and the improved quality of each release it appears remarkable from where I stand.

Detonating software is always dangerous. Will it ever come back together again? Were we right about the surface area becoming too large or the architecture verging on too complex? Stay tuned. We’re in the process of blowing up a lot of our software. For example, we’ve realized our past approach to administration just doesn’t scale. Early on we built a nice UI for editing lots of the configuration properties of a Splunk server. But over time our ability to quickly add features outstripped the surface area of the UI. So we’ve been making configuration parameters available in editable configuration files. Now that is all fine and good but it’s not very discoverable and it’s completely out of context with the task at hand when you’re using the product. Definitely a candidate for explosives. Sometime in the near future you’ll see the administrative side of Splunk blasted for a much more scalable, discoverable and in context design we call “search based administration.” This is one small example of how we’re constantly blowing up our software.

Recently we’ve also been lighting the fuse on our business planning process. It used to be we’d have a few days at the beginning of our quarter when each department in the company (sales, marketing, engineering, customer support etc) would get together and have their own planning process. As we’ve doubled in size since the beginning of the year our old way of planning wasn’t working. Despite our completely open work environment (we have no cubicles or offices) communication across groups had slowed to the point where it was causing a lack of effective planning. You guessed it. We blasted it. Started over. Asked everyone what would make for a better planning process. This quarter we started with a full day of conversations. Everyone was invited to run a one hour discussion forum on any topic they wanted. The only rule was you had to publish it a week a head of time and provide a brief description of the topic on our internal wiki. We had 15 discussion forums run by people all over the company. That was it. Our Q4 planning. A bunch of conversations. We’ll see how far it gets us ; )

BTW, I heard someone at Splunk say in response to blowing things up,

“perhaps companies that don’t blow things up often enough end up blowing up themselves.”

Certainly food for thought. I’m keeping my dynamite close by.

I’m Michael Baum. Welcome to my blog.

I hope to find time to write about some of my favorite topics including:

• Splunk and IT Search.
• Technology gadgets and software — the stuff we all like to use.
• Datacenter applications, servers, networks and security — the stuff we all have to keep running.
• Business, entrepreneurship and venture capital.
• Wall street and investing.

Comments are always welcome and you can also reach me via email at thebaum (at) splunk (dot) com.