thebaumblog: Splunk Apps

If Splunk Was An Animal What Would It Be?

Splunk 4 is out of the bag and the Splunk community and our customers are kicking the tires. I even saw several executives from other log management, SIEM and system management vendors registered and attended our world-wide webcast with a thousand attendees. And Twitter is all abuzz with questions, answers and some ass kicking. Yes Splunk 4 kicks ass. It is 2x faster on indexing and up to 10x faster searching. We have a fantastic new App framework where you can build custom views, dashboards and work flows and there are countless numbers of other great improvements and new features. But sometimes we don’t get it completely right and you all let us know.

But back to my question, if Splunk was an animal what kind of animal would it be?

“Odd thing animals. All dogs look up to you. All cats look down to you. Only a pig looks at you as an equal.”

- Winston Churchill

I read that quote today at the birth place of Winston Churchill and it reminded me that Splunk is like a pig. We’ve always looks our users and customers straight in the eye with the good and the not so good. This has always been the transparent way we conduct business. So keep the feedback coming - the praise and the criticism.

One of the areas that I’m especially interested in hearing about is our new App focus. We are in the very early stages of creating Splunk Apps and making them available to the Splunk community. Some are free Apps and some are premium Apps. The free apps are available for immediate download. The premium Apps you need to talk with us about so we can work with you on an installation. At some point we plan to have trial versions of the premium Apps available for download too.

The free Apps include things like

You can easily download the App .spl file, drop it into your splunk/etc/apps directory and check it out. More easily you can download and launch the Apps right from your Splunk Launcher screen (which is an App too). We’re working on fully documenting all these Apps so if you need help now feel free to contact us via support@splunk.com. You can also select “Send Feedback…” on the first menu of the App to contact the specific App team directly via email. We’re especially interested in what doesn’t work, where you get stuck and what else you’d like to see. Several of these Apps are still beta versions so feedback sooner rather than later is much appreciated.

Happy Splunk4ing!

Splunk Lab in Asia Launches to Develop New IT Search Apps

The last two weeks I’ve been traveling throughout Asia with our new partners at Systex and the Splunk Asia team. In Singapore, Hong Kong, China and Taiwan we met with government agency, high tech manufacturing, insurance, online gaming and managed service provider customers who told us how critical Splunk is to their IT organizations, especially as budgets get even tighter.

Systex is now our master distributor covering Taiwan, China, Hong Kong, Singapore, Thailand and Malaysia. Systex is an amazing company fueled by Taiwanese entrepreneurship, creativity and innovation. The company is part distributor, part reseller, part system integrator and part independent software developer. The 2,900 Systex employees are led by CEO Hilo Chen and COO Frank Lin. Hilo did a stint at Yahoo! Asia before joining Systex as CEO. He is a very friendly, engaging and good nature executive who commands the passion of his team. Frank is detail oriented and intense and he has an ability to focus on what seems to be the impossible and get it done.

I’m not used to people pushing faster than I do, but the Systex team are reminding me what start-up speed is all about.

The Systex system integration and software business is fueled by more than 1,400 engineers with deep domain expertise in financial trading and banking systems, network security, database administration, storage, virtualization, disaster recovery, IT service management, telecommunications OSS/BSS, unified communications, business intelligence and more. This past week we unleashed the creativity of more than 400 of those engineers, product managers, sales personnel and business unit heads. We met at a three day kickoff event for the launch of a joint Splunk Lab designed to come up with new areas to apply IT Search and new Splunk Apps for a variety of use cases.

It is our hope that our joint work together will result in lots of new Apps available for download by Splunk users all over the world.

The event started Thursday with a press conference at the Westin in Taipei. We were joined at the press conference by more than three dozen press covering innovation in Asia. We discussed the design of the partnership, the Splunk Lab and some of the joint customers including Allianz Insurance, IAH Games, and The Malaysian Prime Minister’s Office. Allianz is using Splunk to report on F5 Big IP load balancer activities. IAH is mining their online multi-player game events and logs for insight into user patterns and activities including market basket analysis across different game properties. The Malaysian PM’s office uses Splunk to secure their email messaging system.

The press asked some very good questions about various use cases and our strategy for accelerating activities in Asia with Systex. Richard Tang and Johnny Lin attended the event from Systex as well and provided a great overview of how the Splunk Lab is coming together and what kind of solutions Systex is creating around Splunk. Richard has been very patient with me and has taught me enough Mandarin to completely embarrass myself during my last few visits.

On Friday 260 engineers and product managers attended an all day Splunk Boot Camp at the Systex UCOM training center in downtown Taipei. The day was divided into two three and a half hour sessions. Each session covered using, administering and deploying Splunk. There was a brief section on developing Splunk Apps including building of a network management application.

One of the product managers commented to me at the end of the day, “My mind is broken on Splunk, there is so much you can do with it.”

Saturday’s session was the Splunk Lab kickoff event and creative activity attended by 300 business unit heads, sales people, product managers and field sales engineers. I was amazed. We went from 8:30am to 6:30pm on a Saturday. The level of energy was unlike anything I’d ever experienced before. Taking the long trip back from Taipei by way of Tokyo, I am just in awe at how two organizations half a world a part have so tightly bonded in just six months. I’m very impressed by the Taiwanese work ethic and dedication.

Kord Campbell, Splunk’s Director of Developer/ISV program gave a great talk on developing Splunk Apps to start the working round tables. Each business unit (twelve in all) spent three hours coming up with ideas for Splunk in their unit including what Splunk Apps they were going to create and which customers they were targeting. The areas included

  • Financial Trading Platforms
  • Banking and ATM Systems
  • Database Serivces
  • Information and Security
  • Business Continuity and Disaster Recovery
  • Customer Service
  • Data Management & Integration
  • Unified Communications
  • IT Service Management
  • Education & Training

Teams were judged on several factors including creativity, feasibility, significance to current business and target customer profiles.

The winning team didn’t use slides but instead acted out their presentation in a 15 minute skit. It was wild and reminded me of how dysfunctional most IT organizations are today. Not that we needed reminding :-)

The Financial Services Business Unit was judged the winner. This team has developed market trading platform software in a joint venture with Reuters and explored using Splunk with their quotes and trading solutions and for market compliance. The first scenario involved monitoring TAIFEX, TWSE and OTC trades and examine patterns indicating potential fraudulent activities.

The second scenario showed how IT Search can be applied to troubleshooting the electronic system including buy side, sell side, cash position, web interfaces, trading systems and risk management. Actors in the scenario ranged from investors, web infrastructure managers, dealer groups, trading managers, CRM users and back office personnel. The team called their solution “A Lighthouse in the Dark.”

Perhaps the most interesting integration of Splunk though was the mining of data from the web application platform to determine which features users tapped into and which ones they tried once but never went back to. By examining page views for new functions and correlating those with trade volume deltas the team can continuously monitor the revenue effects of application and site changes.

The Splunk Lab launch has us thinking about how to get other people collaborating to build new applications for IT Search. We’re planning to launch a public site soon that will allow domain experts from all over the world to work together and create great Splunk Apps. So we decided to take the elevator to the top floor of Taipei 101, the world’s tallest building to look for more…


Top Floor at Taipei 101


View to the East of Taipei

Press Conference


Frank Lin, COO, Systex


Me


Robert Lau - Splunk & Emy - Systex


Hilo Chen, CEO, Systex


UCOM Technical Training Center

Kord Campbell - Splunk


Splunk Lab Team Competition


Winning financial services App


A little bit of fun

Taipei 101 - World’s Tallest Building

Splunk Developer Camp 2008

It’s Sunday night before the start of our first ever Splunk Developer Camp. Never before have we invited developers from our community at large to participate in sharing their ideas about building Splunk Apps and learning about all the cool stuff in our upcoming releases. I think I can speak for everyone at Splunk when I say we are truly amazed with the level of interest and participation. We’ve had to move the venue three times now to accommodate the growing list of participants and while we initially expected the mix would be mostly existing customers, we’re really pleased with the mix of developers coming tomorrow.

  • 125 Developers
  • 91 Organizations
  • 26 Industries
  • 9 Countries

Only a third of the developers showing up are customers. The rest are system integrators, MSPs, OEMs, ISVs and VARs.


Post Camp Update

We’ve organized the day into a combination of an un-conference format with developer round tables, sneak peaks of future versions of Splunk, demos, demos, demos from customers and partners and training on the Splunk API and SDKs. Our goal for the day was to both educate campers on how to effectively build Splunk apps and to get everyone jacked up about the possibilities. We broadcast the sessions live on Splunk TV.

The day started with a quick intro by me. I gave everyone a brief Splunk history lesson of the past five years and demos of the Splunk for PCI and Splunk for Server Virtualization applications. I wrapped with a discussion of our strategy to seed Splunk everywhere and to enable developers to distribute their applications to Splunk installations around the world in the near future. More on this in a future post.

Erik Swan and Rob Das, my two co-founders followed with a more in-depth evolution of Splunk chat which many focused on all the weird prototypes and company names we thought of before the real Splunk. Some of it is funny and some down right scary. Amazing what guys out of a job can come up with.

Konfabulator Follow Along

Next up Kord Campbell, Director of our Developer Program gave an overview of agenda for the day and reviewed how to register with the Konfabulator and follow along with the many demos up on our SplunkLabs EC2 server at Amazon Web Services. This worked great as everyone could build and run the demos on their own EC2 instance. Kord also showed off the new Splunk Wiki for developers and application users. We’re in the process of moving all our documentation to the wiki as a one stop shop for information on using, administering, deploying and developing for Splunk. A few other Kord matters included the review of our new Developer Program additions including a 2GB Developer Enterprise License for registered developers.

Splunk Apps

Jef Bekes, our Head Designer and Raffy Marty our Application Product Manager then gave a very inspiring talk about the future of Splunk and Splunk Apps. The basic point being in Splunk 3.3 today there is no sense of application context. This means the same default user-interface for all applications and that all knowledge (saved searches, alerts, reports etc.) is shared across all installed apps. It’s impossible also to “switch” from one app to another. Splunk 4.0 attempts to address this whole problem by making applications first class objects that can be containers for collections of other objects at the interface, knowledge and configuration layers. As more an more Splunk applications arrive on the scene this encapsulation becomes increasingly important. Jef and Raffy showed a sample Splunk 4.0 Help Desk application that included custom branding, restricted task-based navigation and structured search user interfaces and results views. Other Splunk 4.0 features were reviewed too; Splunk Web gadgets, the Application builder, improved charting and content grouping.

Developer Platform and API

The Splunk Developer Platform futures was up next with Tom Donahoe, Splunk Product Manager and Johnvey Hwang Lead UI Developer. Topics included the Splunk 4.0 improvements like Application Builder, REST API Additions, UI Extensibility and SDK Support. The Application Builder eases application creation and packaging dramatically improving the experience beyond where Splunk 3.3 currently stands. The Application Builder will be available in both command-line and GUI to provides application configuration isolation and leverage file system security controls. Johnvey reviewed with us planned REST API additions for 4.0 like

  • Alerting: history, status, improved generation
  • Notifications: email, RSS
  • Search scheduling management
  • Knowledge management
  • Authentication: users, roles, single sign-on
  • Distributed: topology data, server metrics

Splunk Ninja

The Splunk Ninja (aka Michael Wilde) graced us with a visit and showed off his demo Godness with a Zero-to-Lightspeed set-up and data eating with the new Splunk Crawl feature in 3.3. Sweet!

Search Language

David Carasso, a Senior Developer and Alex Raitz one of our Solution Architects did a fantastic overview of the Splunk search language and ran through some really cool examples of powerful stuff like

  • What’s the most important hard disk error on each of my hosts?
  • Who sent me the most email?
  • How long do users stay on my website?

David showed us how to create our own search commands too. Awesome stuff.

Large Scale Reporting and Summary Indexing

Steven Sorkin, Head Indexing Geek led a wonderful talk on large scale reporting using great examples like finding violations in security data on application layer firewalls and routers. He covered how we use map/reduce models to summarize batches of events - what we call summary indexing. It turns Splunk into a sort-a time slinky.

REST/ATOM API and Splunk Gadgets