On a crisp and wet Wednesday morning, the W hotel played host to SplunkLive Atlanta 2010. 100 attendees packed the well-heeled conference room, all eager to learn more about Splunk, get burning questions answered and connect with peers.

As with other SplunkLive events, we invited customer speakers to share their experiences with Splunk. On this occasion we had three speakers and every one of them was stellar.

John Daniely – Atlanta Journal Constitution

The first speaker was John Daniely, from Atlanta Journal Constitution, the only major daily in Atlanta and somewhat of an institution enjoying a daily readership of 2.3 million visitors.

John and his team are responsible for all aspects of network security, including Active Directory auditing, security event monitoring, anti-virus, firewalls and URL filtering. Their environment includes several hundred Linux, Unix, Windows servers and 1500 workstations.

John’s presentation was choc full of examples and use cases using Splunk for security. Here are a few of them:

Tracking Conficker

One of the first feeds into Splunk was from their Intrusion Prevention Systems logs. This enabled John and his team to track the recent ‘Conficker’ worm – see the systems impacted and correlate this data with their firewall logs to quickly see if infected traffic made it into the network.

Protect against Malware

More recently, John and his team started sending Splunk SNMP traps from anti-virus software running on individual workstations. John said “when new malware is detected, all we need to do is type the name of the malware into Splunk and boom all the machines that have it pop up on one screen”.

Splunk + SIEM

John also talked about Splunk complementing their SIEM, because of its ability to index any type of data from any source much faster and without building custom adapters. “Many SIEMs require time-consuming customizations”.

Other current and future use cases include using Splunk for change monitoring, supporting Payment Card Industry (PCI) log retention requirements and to start Splunking application log data.

When asked to sum up, John said, “Splunk is very fast and saves us a lot of time. Investigating from one place with the ability to cross correlate between logs eliminates a lot of manual work”. He then offered advice to people evaluating Splunk, “anyone who’s on the fence about Splunk, I recommend go ahead and install it and you’ll be impressed”.

Tim Metz – Cox Communications

Tim is a six-year veteran of Cox Communications, the third-largest cable entertainment and broadband services provider in the country. Tim’s focus is network security, including both enterprise and their high-speed broadband network.

Their environment has “every piece of gear from every vendor you can imagine!” – it’s heterogeneous and geographically distributed. Tim’s strategy is to get as many syslog data sources in Splunk as possible.

Tim’s a big fan of syslog and believes “if you’re not watching your syslog, you’re not watching your network”. He went on to say, “We’re using Splunk for more than security … if somebody has a router or switch, a Linux server, anything, we encourage them to point their syslog at Splunk, which makes it easy to troubleshoot outages and examine firewall logs all from one place.”

Splunk in the NOC

Cox has Splunk running in the NOC and although Tim claims not to be a developer, he single-handedly created the main Splunk dashboard in 1 day from the ground up! He thanked the Splunk docs team for that one. Here’s the screen shot:

For Tim and the NOC team, Splunk helps them fill knowledge gaps between existing “fancy tools”, which provided a narrow view of their logs. The NOC dashboard built on Splunk, provides them with an actionable summary view of critical information, such as the type and #errors for different time periods (last 10 minutes, last hour, same time yesterday).

Splunk in the SOC

Tim also has Splunk running in the SOC, to conduct security investigations quickly and on demand – 4X faster than before. One SOC use case is monitoring for SNMP offenders – was it customers? Or was it their own tools that were misconfigured?

They also found Splunk useful for analyzing usage patterns – to help them find time windows when they could perform major upgrades without impacting users.

Splunk also helps Cox meet their PCI requirements. For example, Splunk monitors for use of the ‘switchuser’ command on highly-sensitive machines and maps user details to use of the command – this lets them quickly filter out routine actions from anomalous actions.

And what about the future? Tim said it was all about new devices and more data. He’s happy with Splunk’s ability to eat syslog at the rate that it’s sent, “The great thing about Splunk is that it’s already ready for new data types and sources.”.

Joseph Rinckey – BlueCross BlueShield Tennessee

Joseph is the lead VMware systems engineer for BlueCross BlueShield Tennessee. BCBST currently serves 4.3 million people across Tennessee and is part of a nationwide association of healthcare plans.

They are relatively new users of Splunk, using it to manage their VMware environment. This consists of 470 VMs, the majority of which are Windows, on 32 hosts, in 4 separate clusters spanning 50 different datastores – this covers about 43% of their total environment.

Before Joe, management’s perception of virtualization wasn’t all that great. They basically didn’t see it as production ready (due to historic issues). Then came Joe. Armed with a determined team of 2 (including himself) and Splunk, he turned this perception completely around, by maintaining uptime and reliability, and building confidence – “We don’t have a downtime on the hosts now with Splunk”.

Now management wants to virtualize 90% of their total environment in the next 12 months!

Choosing Splunk for Virtualization Management

When asked by an audience member why he chose Splunk, Joe’s answer was simple “Splunk does more”. He went on to describe their evaluation process: “We looked at 8 other VMware third-party management tools, including big names, with some nice features, but they were too narrow and specific”. He said, “We wanted more than performance management, we wanted to consolidate our logs, to grasp what’s going on in our environment. We wanted to do more than just get notified, we wanted to be proactive.”

Getting Data Into Splunk

Currently all VMware infrastructure data is being sent to Splunk, including host logs, which includes everything in syslog, plus several logs not in syslog, such vpxa and the hostd logs. He tipped the audience where they can find more information on getting non-syslog data into Splunk:

Joe’s using the Splunk for VMware App, which enables him to grab all vCenter logs in near real-time – an impossibility before Splunk due to volume and frequency, and also vmware.log files.

The end result is that they now have a clear understanding of their baseline environment:

• They know when/where something attached to our environment is failing – and can fix it
• Splunk notifies them if certain ESX processes (hostd) run out of memory
• They can visualize data in charts quickly and easily, see what percentage of VMware tools are out of date; analytics on server utilization and network utilization

What’s next for BCBST and Joe? “To do more with Splunk, including sending application logs running in the VMs to Splunk in order to provide a more complete view, creating management reports that provide greater visibility and best of all, enjoying how much Splunk makes me look like the Jason Bourne of Virtualization”!

Final Thoughts

What struck me about the event and other SplunkLive events I have attended is the collaboration between attendees. When one person asks a question, it’s often another customer attendee that jumps in to answer. This is why we are focusing more than ever on community at Splunk, including the return of Splunkbase, Splunk Answers and our first ever User Conference this year!

Orlando is the 5th SplunkLive of 2010 (following events in Boston, London, Vienna and Munich) and the first ever in Florida. The event drew a capacity crowd of enthusiastic customers and users.

As usual at these events, we asked customers to stand up and talk about their experience with Splunk – how it’s used, where it helps, lessons learned and the impact on their organization. On this occasion, we had two great speakers from Voxeo and Presidio.

RJ Auburn – Voxeo

Voxeo is the world’s largest provider of Interactive Voice Response (IVR) services, supporting over 82,000-hosted ports globally and also have hundreds of on-premise deployments. Over 100,000 developers use Voxeo’s platform to integrate with their existing web applications and communications via traditional, next generation, or social networks – instant messaging, twitter, facebook, skype, SMS, voice, etc.

RJ is CTO at Voxeo and responsible for bringing Splunk in to help fulfill his mission to “make communications simple”. When asked what they use Splunk for, his response was simple, “what don’t we use Splunk for!” And indeed, Voxeo showcases multiple use cases for Splunk. More on that later, but first what does Voxeo’s IT infrastructure look like?

Logging at the Terabyte Scale

RJ spent several minutes discussing Voxeo’s global infrastructure. Their hosted IVR platform spans 7 datacenters across North America, Europe and Asia Pacific. There are over 2000 servers across these datacenters, generating approximately 1 terabyte of raw log data per day in total. These facts alone, pose significant challenges when seeking to make use of these logs and IT infrastructure data: shipping logs to a central server is not feasible due to logistical, security, regulatory, legal and privacy reasons. Add to this the need to save their data for 7 years, due to compliance and regulatory reasons and Voxeo’s policy of 100% uptime SLA to their customers, and finding a way to better manage their IT infrastructure data looked like a signficant challenge.

RJ starting looking for different solutions and eventually came across Splunk. Not only did Splunk’s distributed architecture and scalability characteristics match Voxeo’s requirements, it also fully addressed the different ways they wanted to use their IT infrastructure data:

• IT operations monitoring in the NOC – ability to see 24×7 dashboards across their entire IT infrastructure, monitor network performance, watch for trends, optimization and capacity planning.
• Troubleshooting IT infrastructure issues – when an issue does, operations teams can pinpoint root cause very quickly from one place.
• Providing developer visibility – providing the 100,000 developers on the Voxeo platform easy and secure access to the hosted platform logs, supporting a multi-tenant, scalable approach.
• Meeting security requirements and creating reports to meet compliance mandates – providing the ability to provide visibility of all security-relevant data and also to meet compliance and regulatory mandates, such as PCI, SOX, HIPAA, ISO 17799 and Gramm-Leach-Bliley.

Splunk’s distributed architecture is deployed across all Voxeo’s datacenters, providing secure and rapid access to logs and IT infrastructure data, whilst avoiding the need to ship data around.

Splunk’s scalability model is based on MapReduce, which scales linearly across commodity servers to absorb the growing transaction and data volumes. Splunk also integrates to Voxeo’s single sign-on architecture to provide a seamless experience for external customers and developers using Splunk. Voxeo makes Splunk’s ad hoc reporting as a value add capability embedded in their hosted offering.

More recently, Splunk is also embedded in Voxeo’s on-premise product and integrated into their management console (Prophecy Commander). Providing a replica of the hosted architecture, but for an on-premise environment from a single laptop to a large datacenter.

Final note – RJ’s complete presentation delivered at SplunkLive Orlando is available at the following link (thanks RJ!): http://www.slideshare.net/voxeo/logging-at-the-tb-scale-voxeo-at-splunklive

David Winters – Presidio

Presidio, Inc., is a diversified professional and managed services firm and recently merged Coleman Technologies, a leading IT and systems engineering firm, providing, amongst other things, information technology and systems engineering services, “We manage outsourced NOCs”. Their NOC environment includes Linux, Windows and Cisco equipment for unified communications.

David joined Coleman Technologies 5 years ago, heading up their managed services group and specifically building the NOC practice. Here’s his version of events, “if you complain enough, you eventually get responsibility and I ended up running the NOC!”

Reigning in the NOC

David’s immediate pressing issue was in helping manage the data deluge. David used Zenoss in the NOC for fault performance monitoring, “Zenoss is great for displaying row-by-row information on the screen, like SNMP traps, syslog and threshold alerts, but the screens didn’t scale as the NOC operations scaled. They found that as they added more customers, more devices, more systems and more advanced technologies, important things simply got pushed off the bottom of the screen.”

He said, “we simply did not have the physical real estate for eyes on glass, to see all the important messages and see what’s going on. This is a big problem”. David and his team then deployed Splunk to manage the low level, high volume data and find problems, which can then surfaced via Splunk dashboards to the NOC.

“Splunk Makes Me Sick to My Stomach”

Let’s explain this somewhat controversial statement. In David’s words, “when I started Splunking the data and seeing what we were missing using our traditional fault performance systems and how we could correlate it and show it dashboards, I literally went home sick to my stomach, not being able to sleep – and then incessantly began using Splunk and finding the silliest errors there were vastly widespread in customer environments – fans and routers that were stopping, duplex mishmashes, VLAN tags that were incorrect. Easy to fix problems that nobody knew about!”

Bringing Important Messages to the Forefront

David and his team sees Splunk to filter out noise and map severities to different message types from custom and packaged applications. Lower level events are Splunked and now David is now able to catch critical issues as they are building – see the frequency of the issue occurring, how many locations it’s occurring at, a break down by field extractions and line of business. By doing this, David and his team obtain actionable intelligence they can respond to quickly. He really liked how level 1 NOC operators can create custom dashboards for specific customers to monitor known issues and without involving development teams.

Final word? Even power users of Splunk get value from Splunk Live! events. David said that after learning more about dashboards in the product demo, he built three of them during the session!

Yes, there are plenty of avenues to get free help on Splunk:

* peruse the online manuals
* watch a video
* enroll in a Splunk training course
* call/email/submit a case with Splunk Support
* post questions to Splunk Forums
* pose questions to Splunk Answers
* gear up on best practices on the Splunk Wiki
* attend a Splunk Live event and bring your questions in person
* compliment your Splunk sales engineer

Back to my point. In a fraction of the time required to take full advantage of the vast resources available (see above), hurdle the learning curve *and* deploy Splunk in your spare time, a Splunk Solution Architect can get you on your way to that next promotion. In the past 3 years, I have had the great privilege of working directly with many Splunk customers. The happiest customers are those who get up and running quickly with a Solution Architect.

Splunk Solution Architects are highly skilled in configuring, tuning and scaling Splunk. They have completed hundreds of Splunk engagements around the globe, some were probably done in your own backyard. Their experience integrating Splunk with home-grown/packaged technologies is unrivaled. They have seen all manner of architectures, and can balance best practices with project constraints and requirements. They could probably finish an implementation blind-folded with one hand.

Not only can they stand up an implementation of Splunk effectively, Solution Architects can provide guidance towards the most sensible architecture and system specs for your environment and needs. They can work with your team one-on-one to scope a deployment. They can customize Splunk by building searches, alerts, reports, dashboards. They can even fast track you from an older version of Splunk to the latest. Solution Architects have both deep product knowledge and the experience to build holistic solutions. While they are absolutely wonderful people who like to help, Splunk support engineers and sales engineers are transient, tactical resources. Engaging Splunk Professional Services is the most effective way to realize return on investing in a Splunk license.

If that doesn’t convince you, if I may, I’d like to tell you a story. I woke up one morning and could barely move my right hand. For months I lived with constant pain opening doors, combing my hair, trying to use scissors, even typing. I took the pain everywhere–to meetings, on the road, through airport security, on long walks. Early on, I consulted my sister the doctor and conferred with her doctor friends, who all told me to wear a wrist brace and simply stop using my hand for 3 months. At first, I saw it as an opportunity to become ambidextrous, and even enjoyed making up stories about how I was injured. After a few days, I grew weary of answering questions so I stopped wearing the wrist brace. The brace would go on or come off depending on how I felt about answering questions on any given day. The pain never went away.

For no less than 8 months, I hobbled along with a busted wrist, harvesting collective wisdom from well-meaning folks on carpel tunnel, massage therapy, acupuncture, X-rays, fractures. Then finally I went to visit a real doctor. The Professional asked me a series of questions. She pressed gently at various points on my wrist and asked about the type of pain produced. She reviewed the potential causes. Most importantly, The Professional told me to wear the wrist brace at night while asleep for 2 weeks. In only a few days, the pain had improved significantly. Two weeks later it was gone completely.

I could have saved myself months of pain, frustration, struggling, wondering what’s wrong, and believing all the free advice could stabilize it. Sound familiar?

Just give a Solution Architect a few days or better yet a week or more, and in return they will give you the keys to an intelligent and performant Splunk deployment. At which point, you can focus that laser intensity on solving problems for your business instead of wrestling with regex and wondering what goes where in which .conf file. Engage a professional and get better faster.

Due to this inherent capability, Splunk can index data that is not necessarily meant for consumption by IT staff and has more of a business focus. Much of industry specific business related data lies within unstructured files or locked in proprietary applications that only allow access via an API. If this data were indexed, it would solve a problem of effective storage, time based retrieval, and ad-hoc search. That’s just the tip of the iceberg. The lens of Splunk can be used to do more with this ocean of data. If we add functionality that includes statistical analysis, aggregate reporting, and alerts, the business value of the data increases to the point that it may effect the bottom line of the mission. All this is possible with Splunk.  Because Splunk does not rely on a database to a priori construct a schema based way to store this data, it makes it tremendously easy to index it and provide the rest of the capabilities just mentioned. Rather than continue to pontificate on the virtues of universally indexing business data, I will provide more concrete examples through my own experiences, which should shed further insight why this is a worthwhile endeavor. I’ll use examples from different industries to illustrate the topic.

Financial Services

One of the most common use cases with Splunk in Financial Services has to do with monitoring trading systems. A trade logically can travel from front to middle to back office to payments in its processing history. All the information about the trade’s activity usually ends up in file after file on different servers. Splunk could be used to index this data to access it from a central console.

The first use case is where is the trade in the system? As it travels from one place to another logically or physically, a Splunk user could simply type Trade=”Some ID number” and instant results come back. With a little more effort, a Splunk power user could create a form search view where the trade ID is entered and the results come back in a more tabular manner that is consistent with what a business user would like to see. Trades may be rejected for a variety of reasons beyond technical difficulties such as incorrect CUSIP, commission rate not supported, or insufficient funds. Knowing this information ahead of time with alerts or ad-hoc searches can provide a quicker resolution to what is adversely affecting a trade. Here’s a form search that I use with fictitious trade data for a demo.

Trade Form Search

This opens up a number of possible use cases in the middle and back office:

• A report on the number of accepted, in process, and rejected trades.
• Alerts for rejected trades for high amounts.
• Summary of the currency amount traded per day
• Summary of the amount of securities bought or sold
• Average number of trades per account on a given day report

The possibilities for business activity monitoring are only bounded by what data the system produces. Instead of spending a large amount of resources on building a sophisticated trade tracking system, which essentially can produce the same results, Splunk can easily be used to monitor trade activity that is posted in unstructured text files. Splunk users are currently indexing this type of data for their business needs.

Stocks

To continue my demo example, if you are monitoring trades, then it makes sense to also have the ability to monitor stock trends which may correspond with the trading activity. As part of my demo, I also use a publicly available web service to call as scripted input where the output of my web service client is indexed into Splunk. Naturally, I am monitoring stock activity on specific securities that happen to correspond to the ones being traded. You can download this add-on from Splunkbase. Here’s an average stock volume graph from my demo using real stock volumes.

Sample Volume Report

News

If the Splunk Dashboard is giving you statistics on trade activity and stock trends, it makes sense to also know what is causing an increase in volume activity in the market. To do this, you can start indexing RSS data into Splunk and show relevant articles on the same dashboard that is showing trade and stock activity. Again, the complete download for this input is on Splunkbase.

Finance headlines

At this point, by universally indexing business data that happens to have trade activity, stock trends, and related news, you now have a Splunk based application that provides business value beyond the raw data for your enterprise.

Telecommunications

There are a number of Telco examples on how people use Splunk with this type of business data, but the one that stands out to me involves records for call events known commonly as Call Detail Records (CDR). A typical CDR event may look like this:

01-10-10 10:55:00, 4153458765, 4153455634, 34343, ...

CDRs are usually in a delimited format where each record has a number of fields separated by an ASCII delimiter. Each field represents an aspect of the call as in caller, receiver, time, duration, etc. This not only makes it appealing to index into Splunk, but also provides a capability to automatically extract the individual fields at search time giving flexibility should the format of the record change.

Because call volumes are so high, this type of data usually involves billions of records over a given time making it difficult to simply put it within a relational database and just as difficult for customer support agents to explain to their customer what were the details for a call made 6 months ago, and why they were charged for a 30 minute call, which they aren’t being charged for currently. Ad-hoc search with Splunk makes this simpler to perform for the call center that needs to investigate billing inquiries.

A demo I like to give with fictitious call detail records to is to show an aggregate report on what types of calls are being made in a given time period so that a cellular marketing department can decide what types of promotions to run at that time. Here’s a sample dashboard depicting this:

Calls By Type

Another use case that Splunk can be used with CDR that is more urgent in nature is about who is calling who and who else did the initial recipient call.  Splunk’s transaction search command can be used to group similar records to provide law enforcement this critical data to carry on their work. On a lighter side, this same search result can be used by marketing agents to up sell friends and family network plans based on who calls who most often.

Environment

My last set of use cases involve situational awareness at the physical environment level. Last year, someone who maintains a high technology building, asked me if it was possible to index building statistics via an API. The answer is, of course, yes. Splunk would call a script on set intervals to call their API to get their statistics and the standard output of that script would be indexed into Splunk.

Building sending data to Splunk

The statistics centered around the environment of the building which included temperature and humidity readings for each floor. Keeping in mind that this turned out to be a hypothetical use case, nonetheless, the implementation for it can be real. Splunk can index each floors’ readings and then provide trend analysis for temperature and humidity that can be correlated to costs. Moreover, if an API is provided to control roof and basement fans, alerts could be raised via Splunk when the medium temperature reaches a threshold that would trigger fan activity. Similar alerts could be used to monitor hallway lights, fire alarm responsiveness, and sprinkler water levels. This may sound like a far out into the future use case, but the technology to implement it is here today.

Weather

A more concrete use case in the environmental arena is indexing weather reports for different cities, which is another demo I sometimes present to users that want to see the different source types that Splunk can handle. As in the previous use case, Splunk calls a web services API every few minutes to get the report on a list of cities provided to monitor. The weather report for each city comes back in XML format. Once again, you can download this distribution from Splunkbase. Below are some sample Splunk reports that can be derived on the fly from this data.

Maximum Temperature

Average Humidity per City

The business impact for this type of data is not as high as in the previous use case, unless you work in meteorology or are planning a trip, but the data itself can be used in a larger context. If you are monitoring climate changes, indexing this data for further analysis would assist in ascertaining climate impact on external forces such as fossil fuel emissions. If other types of scientific data are also indexed into Splunk, correlations or lack of correlations can be made. Splunk will not solve the climate debate, but the task of universally indexing, searching, and reporting on this data provides insight that may otherwise be laying in generated files or locked systems.

Conclusion

I hope this non-exhaustive list of examples has provided an entertaining and informative vision for some types of data that can be indexed into Splunk. Splunk users are already indexing trade data and CDR events into Splunk today. As time moves forward, the breadth of what is being indexed by actual customer use cases will increase beyond what we imagine today. Splunk as a technology shows that not everything needs to be force fed into a database to perform analysis. As most of the world’s data is not actively stored in a search-ready manner, this opens up an opportunity to solve a problem for real business needs without having to retrofit unstructured time series text data into structured containers. If you have ideas on indexing other types of business data into Splunk, please let us know, as deriving business value from your data is a worthwhile investment.

Splunk competed successfully in the morning, drawing a room full of interested Splunk Live! attendees despite the brand new BMW cars and motorcycles on display in the BMW-Welt entrance.

Mika Borner: Swisscom

The first customer presentation was by Mika Borner, a long-time Splunk user. Swisscom is the leading telco/ISP in Switzerland and Mika spoke about their use of Splunk for managing their Internet messaging services.

Before Splunk: custom parsers/analytics, grepping through even one day’s logs took a long time (Swisscom handles 40 million emails per day), there was no live view and finding anomalies was almost impossible. In short, managing the distributed environment was hell.  More importantly, a high percentage of the messages going through their network was spam.

With Splunk: They no longer need custom parsers and can get a handle on what’s really happening in their environment.

“We’ve got a near real-time view on what’s going on, adapting for new logfiles is straightforward, and searching and alerting about anything is easy.”

“Think different. The only limit what Splunk can do for you is yourself.”

Swisscom uses Splunk for troubleshooting and investigating user and infrastructure incidents. Finding and preventing abuse and fraud–including preventing phishing emails, and abuse and fraud of their SMS service–was the initial driver for purchasing Splunk. They were further able to justify the purchase of Splunk to address service crashes. Not only did Splunk greatly reduce the time to resolve issues, they achieved ROI almost right out of the gates. Splunk is also used for reporting, statistics, trending, and capacity planning.

Splunk is used to monitor, analyze and report on Swisscom’s Internet messaging.

Mika created simple form-based searches, enabling Tier 1 people to easily find the data they need, such as all internet messages sent by a specific email account over a selected time period.

The Swisscom Splunk deployment consists of 2 Splunk indexers, 1 search head, capturing 140GB/day, and storing 6 months of data on a 10TB SAN. They use Splunk forwarders whenever possible, and make heavy use of Splunk’s Common Information Model.

Near the end of his presentation, Mika said: “How would I describe Splunk? Eierlegendewollmilchsau” (loosely translates from the German as an animal that does everything!).

Alexander Strobl- Accenture Technical Consultant

Alexander gave a presentation detailing how one of his clients uses Splunk. The client is one of the largest worldwide trading and services companies, with more than 50,000 employees on three continents. Before Splunk, the company was often faced with critical service downtime—a common problem for retailers both online and off.

Alexander said that now, with Splunk, “In 15 minutes I can end all the finger-pointing.” They keep tabs on the general health of their environment using Splunk dashboards, and Alexander recommends, “Wrapping your processes around Splunk to uncover its true power and benefit.”

Splunk is integrated into 10+ business critical applications and services, generating 20-50 GB/ day or approximately 1200 events per second, including custom files and events. The current deployment consists of 2 Splunk instances—one for testing; one for production, with data from hundreds of servers including WebLogic and custom Java logs. They’ve established interfaces between Splunk and other tools to speed problem resolution and issue trouble tickets.

What’s next? The client is planning to use Spunk for transaction tracking across all apps and services, and doing business process analysis.

To celebrate the first German SplunkLive, Splunk Sales Engineer Christian Glatschke marked another first–the first time a Splunk product demonstration was given while wearing Lederhosen. Thanks to all who joined us Next stops in EMEA—Stockholm and Amsterdam in early May.

What I was unaware of until this article came out was the following: “Experts say the underground secondary market where hackers buy and sell stolen online gaming accounts, items and in-game currency has become a billion-dollar criminal industry. In hacker forums, a WoW character account can sell for as much as four times the value of a stolen credit card, said Steven Davis, chief executive officer of game security firm SecurePlay.”

If this sounds like a case of ‘art mirroring life’, it hit me that way too. In real-life, identity theft occurs and for a time these stolen identities were bought and sold in a sort of hacker market place. The interesting difference for me is that every action in the virtual world leaves digital footprints in log data where in real-life this isn’t always the case. This points to a need for a very highly scalable solution that can provide for monitoring of user actions while looking for patterns of account activity that could mean identity theft or fraud in the game.

Because the types of fraud/threats to players are constantly evolving, this isn’t a situation where a filtered SIEM style view of the games logs will work. Detective work can’t be limited or filtered to only what you expect to find. If the fraudsters limited their fraud attempts to what was expected – my guess is that we’d have stamped out fraud a long time ago. No, what’s interesting is what you don’t expect to find. Imagine a CSI episode where the hero limits the investigation only to what they expect to find. This would be bad detective work and boring television.

With over 10 million players in the game (that’s the total population of New York City and Chicago combined), the bad part of art mirroring life will continue given the amount of opportunity. Policing a virtual world can’t be easy and with all the players thinking that everyone in the virtual world is there just for an innocent bit of fun, thieves are likely much more emboldened and opportunities too huge to resist. Massively scalable search against all the data for patterns is Splunk’s forte.

Pollution can happen for a number of reasons:

• the wrong timestamp is extracted (events are dated in the past or future)
• events are broken at the wrong place
• incorrect metadata (host, source, sourcetype) is associated with an event

What does this mean to you? Pollution can cause time-bound searches to return inaccurate results. For example, if you are searching over the last 24 hours and events are incorrectly dated a week ago they will not be returned as part of the result set. Any subsequent operations (e.g. stats, timechart) on the result set will be inaccurate. Pollution can also cause skew in the event count. If Splunk inadvertently breaks an event into multiple parts, the reported event count will differ from the true event count. Thirdly, if the wrong sourcetype or host data is assigned to an event, searches on sourcetype or host will be troublesome.

What can you do if any of the conditions above threaten the integrity of your Splunk installation? It is possible to delete events, whereby they are not returned in search results, but even delete has its limitations. The alternative is to clean and re-index data. This is a very heavy-handed approach and assumes you do not mind losing/reprocessing many millions/billions of events or months/years of data.

Preventing pollution is the best policy. Problems can easily go undetected in a sea of events. Ensuring these problems don’t crop up over time when they become more difficult to address can save you time and save you from having to make difficult decisions about re-indexing.

Here are some simple ways to help you defeat contamination:

1. When first setting up Splunk or adding a new data source, run through some safety checks to make sure Splunk is indexing the data sensibly. Check out the attached on-boarding checklist for some suggested sanity checks.
2. For testing, use a staging environment, not your production Splunk installation. Get a sample of the data and see how it performs. Use Splunk Free, use your desktop, use your neighbor’s desktop–anything but the production Splunk server. If no alternative to the production server is available, at the least, setup a sandbox index where you can test the new data to your heart’s content. When you’re done testing, divert the data stream to the default index (or wherever you need it to go), then delete the sandbox index. Cleaning an index is much easier than trying to surgically remove events from an index.
3. Remove the guessing from timestamp extraction, line breaking, sourcetyping. For your convenience, these 3 topics are covered separately in my previous blogs.

Pollution is not our friend.

First we’ve got SplunkLive Munich. Monday, March 8, 2010 at BMW Welt.

Christina Noren VP, Products, and Steve Sommer, VP Marketing, will be representing the Yanks, along with our new German crew, who’ve just opened our Munich offices.

Alexander Strobl has been bringing the power of IT Search to Accenture’s enterprise clients in Germany where he works as a Technical Consultant in the Data Center Technology and Opeations team. Alexander is responsible for analysis, design, roll out of Splunk. His most recent Splunk project was with a large worldwide services company with more than 50,000 employees on three continents operating mail order, distribution, e-commerce and over-the-counter-retail trade. Accenture implemented Splunk to transform the management of several technologies including Linux, virtualization and large-scale storage systems.

Then Mika Borner, Head of Internet Messaging for Swisscom will tell us how Splunk’s monitoring, alerting and reporting helps to keep its network running in peak form and helps Swisscom to fight spammers and e-mail system abusers. He first heard of Splunk when we held SplunkLive Zurich in 2008, and now he’s back to share his own success story.

Register now for SplunkLive Munich to join the discussion and see the latest Splunk features.

Back on US soil, we’ll be attending SANS 2010 in Orlando, FL at the Swan and Dolphin on March 9. We recognized as a User Vetted solution to address SANS Top 20 Critical Controls–check out why in booth 107.

And if you’re in town for SANS, why not swing over to SplunkLive Orlando on Wednesday, March 10! We’ll be just down the street at the Sheraton Safari Hotel and Suites.

For the SplunkLive event, long-time customer Voxeo takes the stage to share its success. They help enterprises improve service and lower costs by automating and connecting their most common phone calls with its Interactive Voice Response (IVR) or Voice over IP (VOIP) solutions. More than 100,000 developers build apps on Voxeo’s platform, and they access the data they need to troubleshoot those apps through Splunk! Plus more than 150 staffers in Voxeo’s 24 x 7 NOC watch Splunk dashboards to watch for spikes and errors, then dig in to remediate problems before they cause network outages. They are power users of Splunk and it’s a great opportunity to see the places you can take Splunk in your IT environment.

Coleman Technologies, a leading-edge IT and systems engineering services provider, uses Splunk to support the availability, security and compliance of IT systems it maintains for multiple customers. It’s first and second-tier staffers monitor Splunk to keep customer’s systems online and customer satisfaction scores high.

Register now for SplunkLive Orlando to join the discussion and see the latest Splunk features.

From Orlando, we’ll jet up to Atlanta to host Cox Communications, The Atlanta Journal Constitution and a large healthcare provider for SplunkLive Atlanta on Thursday, March 11, 2010.

Cox Communications delivers cable and telecommunications services to more than 6 million customers. Cox uses Splunk to run its NOC, SOC and conduct forensic investigations.

The Atlanta Journal-Constitution is the only major daily newspaper in Atlanta, Georgia. The AJC is the flagship publication of Cox Enterprises and reaches more than 2.3 million unique visitors per day. The AJC gets a single view of its security posture across workstations, servers, network and security devices using Splunk.

The large healthcare provider has virtualized much of their IT environment–hosting critical business applications, development servers, and the webservers hosting subscriber information websites all on VMware instances. As you can imagine wrangling and troubleshooting all of these VMs can present quite the management problem–which is why it chose Splunk to help ensure uptime, and facilitate capacity planning. Join us to learn more about the proactive ways this IT team is managing its cirtual systems with Splunk.

Register now to join us at the W Atlanta – Perimeter on Thursday March 11.

Join us if you can, or send your friends or colleagues–should be great to hear these customer stories!

January 10, 2010 02:10:00
01/10/2010:14:10:00
10 Jan 2010 02:10.00
01-10-10-14-10-00
… and on and on

Splunk has many default rules in place to recognize all sorts of date/time stamps. In most cases, Splunk will automatically detect the right date/time in the event or the file name and normalize it to the Splunk server time. At the least, when adding a new data input, please ensure the date/time Splunk extracts matches the event date/time. The Splunk date/time is displayed to the left of the raw event in dark gray when viewing events in the search view.

However, if Splunk didn’t have to make an educated guess about the date/time, it can index data that much faster. Configuring explicit rules for date/time extraction can make Splunk more efficient. Additionally, while Splunk is quite good at recognizing date/times without any help, it’s impossible to have the default rules cover all possible formats, in which case you will need to configure timestamping rules.

To do so, use these parameters in props.conf:

[mydatasource]
TIME_PREFIX =  <regular expression>
TIME_FORMAT = <strptime-style format>
MAX_TIMESTAMP_LOOKAHEAD = <integer>

These options and examples are documented in $SPLUNK_HOME/etc/system/README/props.conf.spec and here.

Minimally, if you know the date/time appears at the beginning of the event, say in the first 32 characters, then reducing the MAX_TIMESTAMP_LOOKAHEAD will prevent Splunk from reading further into the event than it really needs to in order to find the date and time.

You can also reduce the havoc wrought by the very ugly problems of having events dated erroneously in the distant past or even in the future by tuning these settings, also in props.conf:

MAX_DAYS_AGO = <integer>
∗ Specifies the maximum number of days past, from the current date, for an extracted date to be valid.
MAX_DAYS_HENCE = <integer>
∗ Specifies the maximum number of days in the future, from the current date, for an extracted date to be valid.

The essential unit of correlation in Splunk is time. To make the best use of a time series search engine, getting the timestamping correct is a must, and doing some basic date/time config can also make for a leaner, greener Splunk.

Being at SCALE got me thinking about how much Splunk has grown as a technology and a company. This year, we added two new members to the booth and the Splunk Southwest Team–Jason Stein and Ron Naken. Jason is our Regional Sales Manager and Ron our Senior Sales Engineer. Jason and Ron are well-connected so you may already know them from their previous posts prior to joining Splunk. I can’t think of a better welcome to the southwest than to meet local customers already using and loving it.

This was our third time participating in SCALE. During the first run, we presented Splunk 3.0 with new reporting features. The top 2 questions were “What is Splunk” and “Is it open source?” The top comment was “We can build this ourselves.” On the second run, we showcased a faster, more polished Splunk. More people were coming by the booth to ask questions about setting up and using Splunk since they’ve already downloaded it. This year, Splunk 4.0 was in the limelight with an even more responsive interface, dashboards and Splunk Apps. More people stopped by to tell us they are already using Splunk and how it has changed the way they work.

This timeline holds true for BB, a Splunker at a large Southern California energy company. He is responsible for securing and investigating abuse across a network serving well over 20,000 internal users. Two years ago, BB thought Splunk was an open source tool, requiring lots of consulting to implement and scale. He made a quick pit stop at the booth for a demo and agreed to download Splunk. The second time we ran into BB at SCALE, he hadn’t yet downloaded Splunk, but agreed to do so. After a short, successful evaluation, BB became a customer. This year at SCALE, BB was back to tell us Splunk has been invaluable in helping him troubleshoot problems.

When a user calls in with a problem, BB simply looks at the name of the caller on his phone, executes a Splunk search on the user name, and can immediately determine whether the problem is originating from the desktop or the network. In the past, troubleshooting sessions like these would require him to first prioritize the problem based on its impact on the caller’s productivity, then schedule a web meeting to reproduce it collaboratively with multiple terminals open. These web meetings would typically take 30 to 45 minutes from initiation to resolution of the problem.

Now with Splunk, when a user calls, BB is able to diagnose and repair in a matter of minutes with the user still on the phone. This is just one of the ways Splunk has made his life easier. He has also been able to investigate interesting behavior across the network to detect abuse without relying so heavily on custom scripts which can take hours to complete.

This year, I didn’t hear a single person say they could build a tool like Splunk internally. Several people did share they attempted a similar solution in-house, but it was not nearly as flexible or scalable. There were still plenty of people asking “What is Splunk?” and “Is it open source?” One can start to feel like a robot answering these questions in repetition, but when a BB comes along to share his triumphs with Splunk I am reminded that it’s not just about the cool T-shirts. Splunk’s impact on people’s lives is positive and tangible.

Thank you to everyone who stopped by to tell us how much they appreciate Splunk and to those who came by for an introduction. Keep Splunking, and see you next year at SCALE!