Our first speaker was Andy Purdy, the Co-Director, International Cyber Center, George Mason University and the Former Acting Director, National Cyber Security Division (NCSD) and US-CERT Department of Homeland Security. Andy was a member of the White House staff team that drafted the U.S. National Strategy to Secure Cyberspace (2003) and served on DHS tiger team that formed the National Cyber Security Division (NCSD). He was 3 1/2 years at DHS, the last two heading the NCSD and US-CERT as the “Cyber Czar” of the U.S. Andy is also a Special Government Employee on the Defense Science Board Task Force on Mission Impact of Foreign Influence on DoD Software. He is also a partner with the law firm of Allenbaugh Samini Gosheh, LLP.

The Constantly Changing Threat Landscape

Andy talked with us about the changing threat landscape and lessons learned from past approaches to cyber security that can be applied in a forward looking approach to Risk Management and Compliance.

Since much of his experience has been spent preparing the country for what cyber threats are coming next, Andy thinks of IT security as a war fought in a constantly morphing theater with new technologies and vulnerabilities and new motivations and threats.

A Different Approach Moving Forward

For anyone serious about security this is a sound perspective whether you are a government agency, a major enterprise or a small business. But, the balance between open networks and services and robust security remains one of the major challenges for IT organization. Andy pointed us to lessons learned from his past, fueling a vibrant conversation during the customer and speaker roundtable. Perhaps the most important thing I heard was it’s not enough to prepare for the last war, or the last successful attack. While perimeter defense and legacy standards for network security are provide some measure of security, those measure are very often insufficient to deal with the new threats that seem to be gaining in sophistication at an accelerating pace. Andy encouraged us to focus on adopting new requirements and security infrastructure for situational awareness and control.

Greater sophistication, slower, lower-level attacks, greater knowledge about the targets (data, activity, vulnerabilities) are all contributing to the need for near-time visibility on a large-scale. This has become far more important than sub-second correlation of known attack vectors against discrete sets of network devices.

“NIST perspective: Continuing serious cyber attacks on federal information systems, large and small; targeting key federal operations and assets. Attacks are organized, disciplined, aggressive, and well resourced; many are extremely sophisticated. Adversaries are nation states, terrorist groups, criminals, hackers, and individuals or groups with intentions of compromising federal information systems.”

Andy went on to discuss how the effective deployment of malicious software causing significant exfiltration of sensitive information (including intellectual property) and potential for disruption of critical information systems/services has made detection of inforation and data leakage a key government and enterprise security requirement.

Bob Flores, Former CTO and 31 year veteran of the CIA was our next speaker. Bob retired from the CIA six months ago and is now President and CEO of Applicology, providing cyber security and IT strategy consulting services. In his 31 years at the CIA, he held various positions in the Directorate of Intelligence, Directorate of Support, and the National Clandestine Service. Most recently he was the CIA’s CTO where he was responsible for ensuring that the Agency’s technology investments matched the needs of its many missions. Bob has a Bachelor and Master of Science degrees in Statistics from Virginia Tech.

Quis custodiet ipsos custodes?

Brush up on your Latin! “Who’s guarding the guards” was the topic of Bob’s talk. Insider threat in an every changing threat landscape was and remains our number one cyber security risk.

“Defense-in-depth isn’t just about putting adequate technology in place, it’s also about paying attention to your people and implementing policies and procedures to reduce the likelihood of an insider attack.”
- Dawn Cappell, CERT

The simple but not so obvious model Bob pursued at the CIA was an extension of the ISO stack to include the non-technical but motivational additions.

We need to worry about all levels of the stack including layers eight and nine because we all have people messing around at various layers with applications, scripts, communications etc. And their motivation is often very clear.

Nemo repente fuit turpissimus! Or no one ever became thoroughly bad in one step!”

The point is people don’t just wake up one day and decide to be bad. They are motivated over time by larger causes and in EVERY CASE leave a trail of clues behind that can’t entirely be covered up.

What to Do?

According to Mr. Flores the focus needs to be on real-time visibility. You need visibility into who (or what) is perturbing your enterprise right now and over time. You can tediously review the logs of each device and user as the CIA used to do or you can take advantage of Splunk.

“Splunk may not be the best thing since sliced bread, but it’s pretty darn close.”
- Bob Flores

Why Splunk?

Why did the CIA choose Splunk over so many other security forensic solutions? It all comes down to how easily and scalable Splunk can eat any logs, events and messages Bob’s organization throws at it. Combine that with the real-time search, alert and reporting and over time statistics and analysis on

• user behavior,
• network behavior,
• system and application activities and
• configuration changes

user customizable dashboards to enforce who can see what about whom and full data segregation and access auditing by user or role and you have the answer.



Our last guest speaker was David Duvall, Infrastructure Architect at Discovery Communications. David is a lead technical architect working with teams across four continents to build critical systems and keep them running. Discovery is one of my favorite cable channels. If you haven’t seen it the series entitled Man Versus Wild, is just awesome. I won’t spoil it for you. Check it out. Discovery the world’s number one non-fiction media company with more than 1.5 billion cumulative subscribers in over 170 countries. They run 100-plus worldwide networks, led by Discovery Channel, TLC, Animal Planet, Science Channel, Planet Green, Investigation Discovery and HD Theater. Yes all the good stuff that makes having a cable or satellite subscription service worth while.

We’re Going Public!

And oh we have just 16 Months to show SOX compliance. Discovery went public in September, 2008. The company knew they needed a log consolidation system for retention of at least 13 months worth of data with minimal time for rollout. They couldn’t spend a quarter implementing a new solution. The in-scope SOX environment includes

• 50 domain servers on 4 continents,
• Unix syslog,
• WebSphere app server logs,
• Client desktop logs,
• Network backup status logs,
• WMI Windows event logs,
• Cisco, Juniper and F5 network device logs,
• NetApp filer logs and
• Oracle database logs.

Splunk Deployment

Discovery’s Splunk deployment took 1.5 weeks from start to finish. David was responsible for the installation and personally downloaded and installed Splunk, read the Splunk docs, wikis and got up and running without weeks of services. Most data sources are streamed to Splunk over the network from their native logging facilities.

“I knew I could get Splunk up and running quickly to ensure I captured all the data. Then I could take my time to figure out what I wanted to do with the data.”

Approximately 100 Windows servers were outfitted with Splunk light- weight forwarders to bring Windows event logs, native files and registry change information into Splunk. Oracle database logs are stored in SQL tables and David was able to set-up a scripted Splunk data input which acts like any other SQL client to grab the Oracle database logs on a scheduled basis.

Compliance Reporting Made Easy

Once the initial deployment was complete, David and turned his attention to working with the company’s SOX auditors and department heads to develop the compliance reports required to demonstrate compliance with all the necessary controls.

“As the auditors questions change from week to week—it’s easy to pull new data and generate ad-hoc reports.”

Using Splunk’s role-based access controls, David and the auditors then developed an implemented policies to guard the data and reports including audit reports to prove only the necessary individuals are using the information and to prove authenticity of the data itself. The auditors really like the secure audit trail and signing of data from source of origin all the way through to the Web-based control reports.

Lessons Learned

Adoption of Splunk proved easier than David and the audit team imagined because many of the IT team at Discovery had already downloaded and used Splunk for other tasks.

“When you explain Splunk as “Index and Search” you’re glossing over a lot of the value. Dashboards that correlate failures from different sources and troubleshoot different environmental items are priceless.”

]]> http://blogs.splunk.com/thebaum/2009/09/17/splunk-live-washington-dc-2009/feed/ http://blogs.splunk.com/thebaum/2009/09/16/splunk-live-princeton/ http://blogs.splunk.com/thebaum/2009/09/16/splunk-live-princeton/#comments Thu, 17 Sep 2009 03:50:21 +0000 thebaum http://blogs.splunk.com/thebaum/?p=278 Wednesday and we’re at Splunk Live Princeton, NJ. What an awesome place. Princeton is home to a great university and some great culinary experiences. Check out Mediterra — an interesting mix of Italian and Spanish influences. Apparently it’s where all the Princeton parents treat their kids to dinner when they are in town. Next store to our venue was the great hope for the state of NJ — a new Governor. The current Governor has turned the state budget and tax base into toxic waste. Well things went much better for the more than 60 Splunk Live attendees in Princeton today, who gained insight into how a number of large Splunk customers keep their mission critical applications running in a time of IT budget slash and burn.



Matthew Stevens, Director Software Systems and Architecture at Comcast provides guidance to Comcast executives on mission critical media systems and strategic systems architecture. Comcast is the country’s largest provider of cable services serving 23.9 million cable customers, 15.3 million high-speed Internet customers and 7.0 million Comcast Digital Voice customers.

Comcast Developer Network

Matthew’s latest project is the Comcast Developers Network a Comcast-scale secure web services platform for the development of cool new media and entertainment offerings. The Comcast Web Platform environment generates of billions of software events each day from caching and load-balancing, origin application servers, databases, middleware and content delivery networks for images and video streams. Comcast services demand high quality. Much of the Comcast content is exclusive and premium services drive revenue. Interfaces between technology components (applications, delivery platforms) need to adhere to best practices to ensure the highest degree of end customer experience.

Why Splunk?

Comcast has acquired many system and application management platforms over the years, but nothing was providing the team with the robust information from operational telemetry the teams around the company need to ensure data integrity, stability, application quality and efficiency. Several efforts specifically drove Comcast to consider and deploy Splunk.

• Product rollout: The team wanted the ability to predict and correct potential issues before going live into into production—Splunk has become a required best practice for new product rollouts.
• Network/ System Integrity: Understanding security and user experience across a very large network and set of systems is a must to protect the business. Splunk provides the insight the network and system teams need across many different silos of technologies.
• Business Intelligence: Having immediate access to real-time events and historical trends allows the various Comcast business teams to react quickly and adapt to changing customer behaviors.
• Agility: Alerts and Dashboards indicate discrepancies so distributed teams can investigate immediately and remediate failures and attacks.

Video CDN/CMS Performance

“In content management systems and delivery networks a devil walks the long tail. If you’re facing concurrent hits across the tail of the curve, sharpen your pencil, you’ve got problems!”

Splunk helps Comcast understand the risks of instability in our systems, especially during periods of high concurrency. Through pre-production modeling of even patterns and subsequent monitoring of these patterns Splunk pays for itself by helping Comcast avoid deployment of vulnerable systems, downtime, and upset customers.

Predicting System Imbalance

Comcast has successfully used Splunk to evaluate potential infrastructure vendor’s solutions and determine if they will balance loads properly across a large, indeterminate infrastructure. Often the answer is no as illustrated here in a Splunk report of resource utilization across various services.



Splunk has also been utilized to see whether solutions will be resilient to different traffic patterns, helping the company perform predictive analysis before making critical infrastructure investments.



Load testing is performed during non peak hours and the results are analyzed for system failures over time using the telemetry data Splunk can correlated across various logs, messages and events.



When failures are found the Comcast team uses Splunk reports to dig deeper into the data.




Security and Compliance

In addition to operations use cases, Comcast security and compliance teams leverage the consolidated logs across data centers to enable faster threat assessment and security monitoring.

• Monitoring for bad actors to trigger alerts,
• Conducting threat detection over time,
• Detecting attacks/vulnerabilities in systems and
• Auditing systems in support of security assessments and compliance.

What’s Next?

Next up for Matthew and team is the launch of the Comcast CodeBig Platform enabling a network of developers to create content for the network. Some of these developers are already using Splunk in their own managed services like Mashery. Comcast is working to hook the Mashery Splunk installation to their own in-order to provide visibility across multiple services and providers of content and entertainment functionality.



Chris Abboud manages the Enterprise Systems Management team at Dow Jones — monitoring customer facing infrastructure and applications. Dow Jones provides global business news and information services to millions of consumers and enterprise media groups. Keeping these revenue generating services running 7×24x365 is the highest priority. Chris also manages the DJ service management platforms (Remedy, Knowledge Base, etc.) He’s been with the DJ organization for 10 years, in current role for 3 years.

“Our mission is to address issues before they become service impacting events. Failures are going to happen — we need to make sure people know about them as soon as possible.”

The Splunk Set-up

The Dow Jones Splunk installation includes

• Data from 6000+ servers globally,
• 13,500 + source types,
• 1,700 network devices (primarily Cisco and Juniper) and
• Ten distributed Splunk servers in difference geographies index ~100GB a day and provide a new global logging console.

Why Splunk?

Each Dow Jones command center now has the ability to know what’s happening before customers do across a wide range of internal and external services. Splunk speeds the time to resolution for email outages that may impact internal users’ productivity and editorial sites downtime that can directly impact to customer service and revenue. Dow Jones has found Splunk generates significantly fewer false positives than traditional monitoring systems and new resources are much easier to manage and deploy.

Per server monitoring costs have dropped by a factor of 5X

What’s Next

Next up Chris and Dow Jones will be checking out the Blue Coat and Cisco Apps as they turn Splunk onto those aspects of their infrastructure.



Talk about doing more with less. Andrew Page in the Office of Information Technology at Rutgers University has seen IT budgets go from lean to next to nothing. In this unprecedented time of state educational cuts, Andrew, responsible for enterprise monitoring and service management has turned on and been turned on by Splunk. The self confessed “ITIL guy” at Rutgers, Andrew oversees operations for systems for 50,000 students on campus in three different geographies (Camden, Newark and New Brunswick. The university’s back office supports 27 degree-granting units offer majors in more than 100 fields, with thousands of courses covering the full range of human experience.

The Splunk Set-up

The Rutgers Splunk set up includes

• 2000+ data sources,
• 1,850 network devices,
• ~100 Servers: Windows, Solaris, Unix,
• ~50 J2EE apps
• 5-10 GB logs and messages / day
• 95% coverage of infrastructure in Splunk
• 40+ users

Single Splunk Server

Why Splunk?

Six months ago Rutgers was facing a number of log consolidation drivers including:

• The need for real time access for production logs by service teams,
• Faster cross-silo problem resolution and collaboration,
• Simplification of problem troubleshooting for load balanced applications,
• Decommissioning of “critical” monitoring scripts running in home directories and
• GLBA and PCI compliance and regulatory reporting mandates.



Fast Implementation

Can you fully implement Splunk in a few days? Yes you can according to the Rutgers team. From download through basic implementation took 1.5 weeks and only part of a single resource. The Rutgers implementation included roles for data security, form searches and transaction searches, and custom dashboards.

Performance Management

Andrew and his team use Splunk to grab performance data. A scripted input makes HTTP calls into running JVMs. The team graphs this data and correlates it to load and error messages.



Outage Avoidance

In other scenarios Andrew presented how the Rutgers team finds problems before they become widespread outages. Eight weeks ago a certificate error started causing application failures and could have resulted in widespread outage. It took 6 minutes to answer…

• Who was affected?
• What time it happened?
• What apps were involved?





Lessons Learned

Some valuable lessons from the Rutgers team include and emphasis on distributed deployment and the key to speed of installation. Second, think about security before you start. Third, during deployment get others involved quickly.

We had users on day two. The rule is that if you send in data you get a Splunk account.

Your early adopters will build their own solutions, but make sure you plan for availability as users become dependent on Splunk quickly and will notice any Splunk outages fast, fast, fast.

What’s Next

• Expand use in the Application environment,
• Feed in Oracle databases,
• Migration to Splunk 4, of course…,
• Expanded roles and security around roles should be big win,
• Improved dashboard cache controls and
• Offer some in-house training in advanced skills.
]]> http://blogs.splunk.com/thebaum/2009/09/16/splunk-live-princeton/feed/ http://blogs.splunk.com/thebaum/2009/09/15/splunk-live-new-york-2009/ http://blogs.splunk.com/thebaum/2009/09/15/splunk-live-new-york-2009/#comments Tue, 15 Sep 2009 10:10:39 +0000 thebaum http://blogs.splunk.com/thebaum/?p=270 This week we’re on the East Coast enjoying some fantastic customer presentations and roundtables at Splunk Live events in New York City, Princeton NJ and Washington DC. It’s Tuesday and we have more than 100 customers and Splunk users attending Splunk Live in midtown Manhattan. The vibe is electric as we’re being treated to awesome talks by IDT and New York Life. At lunch, long-term customer’s Bloomberg and AT&T joined the customer roundtable conversation.



Gabe Arnett, Senior Software Architect at Moody’s demonstrated how Splunk is being used to monitor and troubleshoot the Moody’s Analytics platform. Gabe has more than 15 years of building web applications in financial services, investment banking and e-Commerce. At Moody’s he’s responsible for global development team that develops and supports the newly re-designed client facing website – v3.moodys.com. Moody’s is a leading provider of research, data, analytic tools and related services to debt capital markets and credit risk management professionals. The company’s products and services provide the means to assess and manage the credit risk of individual exposures as well as portfolios; price and value holdings of debt instruments; analyze macroeconomic trends; and enhance customers’ risk management skills and practices.

Moody’s Splunk environment is utilized by 25 different users and runs on Windows 2003. Splunk provides Gabe’s developers secure access to the logs they need without touching the production devices, servers and applications. His team has built custom searches and a number of dashboards indicating the general health of their applications and service. Custom searches and alerts provide alerts to track errors and access – guaranteeing good user experience. The team also uses Splunk to understand when and where new content isn’t flowing to the v3 platform. A large part of the Moody’s user experience is delivering email alerts and Splunk helps the team track GUIDs to ensure customers receive the alerts they’ve subscribed to.

The team recently migrated from Splunk 3 to Splunk 4 – taking 30 minutes to perform the upgrade. The Splunk for Windows App has been significantly revamped in Splunk 4 and the Moody’s team is making use of it to monitor through WMI local server resources (disk, memory, networking) and correlate this performance data with the Windows and Application event logs.



Shay Benjamin, CSO and SVP, Architecture at IDTdesigns and implements network architectures and manages compliance, security and fraud initiatives at IDT. IDT Corporation (www.idt.net) is a holding company focused on the telecommunications and energy industries. Since 1995 they’ve been building hundreds of VOIP switches globally and assembling an international fiber optic network. IDT pioneered VOIP (Voice over Internet Protocol) to create Net2Phone, piloted the first commercial WiFi phone service in the US and has created a prepaid calling card business, which sells 12 million calling cards a month.

IDT uses Splunk primary for VOIP Call Detail Records (CDRs). The company indexes more than 120 million CDRs per day with six mirrored Splunk server instances. Call Detail Records (CDRs) are somewhat like logs, but with many fixed delimited fields . One or more CDRs are created at each switching or routing point for every VOIP call. CDRs vary between platform devices in number of fields and contents and unlike logs, few CDR fields contain easy-to-read key=value pairs. Although a key piece of maintaining service quality, billing, monitoring network quality and security forensics, working with CDRs is labor intensive and delay wastes labor, time and money.

IDT needs fast searches across all fields of the CDRs and quick data loading – to allow fast retrieval of call data and cross platform searches to unify results from different CDR formats. Historically IDT utilized a custom RDBMS solution with an application called Call Genius. In their RDBMS IDT was forced to limit the fields that get indexed because indexing of CDRs with an RDBMS is costly as it takes up a lot of space and slows load times. The RDBMS also only indexes fields common to multiple platform’s CDRs. In the RDBMS solution much of the CDR data was put into BLOBs (actually CLOBS) – multiple CDR fields mapped into a single RDBMS field to try and achieve efficiency. But Blobs can be very difficult to search and are difficult to index effectively. The legacy Call Genius application didn’t permit the search of CDR BLOBS.



Now IDT utilizes Splunk to index all CDR fields. No need to decide what fields to index and cross platform searches are easy without losing specific platform CDR format resolution. There is no longer a need to create BLOBs for efficiency. Engineers and support staff are able to quickly search for any combination of

• Phone Number
• IP address
• Trunk Group Name

Splunk naturally and easily links search terms across fields and the users just need to enter the phone number or IP and get back the CDR events and transactions.

Comparing Splunk to the RDBMS solution IDT found searches to be 50 to 100x faster on non-indexed RDBMS data. Indexed fields are also faster in Splunk than in the previous RDMBS solution. Splunk load times for a typical sample average 1 to 5 minutes versus the 20-40 minutes for the RDBMS.

IDT is in the process of feeding firewall, security, router, IP network, and switch data in into Splunk as well. They’re already discovering Splunk is finding errors not captured by Network Management Consoles and has provided valuable troubleshooting during recent datacenter migrations.



Most of all IDT is looking forward to discovering new ways to use all the data in Splunk. Heuristic analysis and Business intelligence applications are on the top of their list including the use of Splunk to find human “Family and Friends” networks and drive the development of new commercial programs.



New York Life Insurance wrapped up the morning session presentations with Aaron Zachko, Assistant Vice President of Information Systems. New York Life’s family of companies offers life insurance, retirement income, investments and long-term care insurance. New York Life Investments provides institutional asset management and retirement plan services. The company has the highest possible financial strength ratings from all four of the major credit rating agencies.

Aaron is a senior network architect and leads the group responsible for network management, core network infrastructure and network security infrastructure. The New York Life network consists of hundreds of Cisco routers, switches, firewalls, enterprise DHCP and Network Access Control (NAC) devices. The company chose Splunk to satisfy audit and compliance requirements and support the rollout of their NAC infrastructure earlier this year. Currently the team is expanding its use of Splunk into enterprise security forensics and as a multi component-monitoring compliment to their Enterprise Service Management Platform which seems to have one of every kind of monitoring tool already.

Thousands of users a day go through NAC to access the New York Life network and Aaron’s team needed visibility into the network from a unified infrastructure and services perspective. They use Splunk to monitor failed login events and transactions and unauthorized devices on the network globally. The NAC rollout team has been able to stay in front of issues – identifying them before end users discover the problems. Their custom Splunk dashboards enable the team to easily see trends and spikes in activity across all networking components.

Operations teams at New York Life have more recently been using Splunk to troubleshoot Application issues.

An application issue across multiple servers created more than 9M events across 167 different sources. Manual investigation into this kind of problem would have taken days — an extremely complex and time consuming effort. Splunk found the issue in 3 minutes. Now teams can trace transactions across systems in minutes or seconds vs. hours or days. And all without any new instrumentation – just using the artifacts they already had.

New York Life is discovering what many other Splunk users have too. Enterprise monitoring and service management platforms can tell you something is wrong but Splunk will help you figure out why and where to fix it.

]]> http://blogs.splunk.com/thebaum/2009/09/15/splunk-live-new-york-2009/feed/ http://blogs.splunk.com/thebaum/2009/08/27/splunk-4-down-under/ http://blogs.splunk.com/thebaum/2009/08/27/splunk-4-down-under/#comments Fri, 28 Aug 2009 05:54:00 +0000 thebaum http://blogs.splunk.com/thebaum/?p=255 I visited Sydney and Melbourne last week to host our first Splunk Live events in Australia. Its my first visit to Australia and I’m really blown away by the friendliness of the people we’ve met. And the “Australian for Grep” t-shirt finally had a proper home. Attendees at today’s event in Melbourne and Tuesday’s event in Sydney included an impressive list of current customers and partners and a number of new users evaluating Splunk for the first time including Telstra, Ericsson, InfoSys, Frontline Systems, Fujitsu, GE Capital Finance, Toll Holdings, Vanguard Investments and more. We owe a huge thanks to the team from Digital Networks Australia who sponsored the two events.



Martin Brown, A Large Australian Financial Services Company

In Sydney Martin Brown, pictured below with me, gave an excellent presentation on using Splunk for Identity Management Compliance. Martin is a Technical Architect managing the development and operations of the world wide web application security system‏ for a major financial institution. He’s had many career evolutions from implantable device electronics and software engineering, UNIX and network systems administration, internet systems management and security.



Martin’s company has a requirement for presenting client security history from their web applications and to be able to access this information to look for suspect IDs from the past six months. Tivoli Access Manager (TAM) is used for both external and internal identity management and access control. More than 200,000 clients authenticate externally through TAM.

His Splunk deployment is very much out of the box with a range of saved searches and some role partitioning. It consists of a single Splunk server with 1TByte of local disk for retention. The TAM logs are rsynced regularly and directly mounted from various hosts and systems. 12 internal and 12 external TAM hosts generate 5 GB/day of data or ~2TB of data a year.



The current user base consists of business second level support teams and TAM support group for third level support. The user bases is expected to extend to the Risk Management Group and first level help desk support soon. Their classic use case is

“Client X’s account has been compromised. What applications has he/she logged in to in the past 6 months?”

The old way required days / weeks of work and support from multiple teams. Often needed to pull in log files from offsite backup tapes then grep through GBytes of data from several hosts. Fun fun. Now with Splunk Martin’s team finds answers in minutes and soon will train Tier 1 agents to do the same, eliminating the hassle of Martin’s team fetching data for everyone. Next he plans to add App server, Web Server and Load Balancer data, role partitioning to restrict business user access to relevant logs, off-shore implementations to present local application logs, API consumption for helpdesk one-stop-shop interface.



Nick Clark, Ericsson

Nick Clark is a Technology Manager in the Solution Management & Utilities Consulting, System Integration & Multimedia practice with Ericsson where the focus is on bespoke support and life cycle management services for complex infrastructures. His group focuses on mobile and fixed network infrastructure, telecom services, software, broadband and multimedia solutions for operators, enterprises and the media industry. He presented his Splunk solution which Ericsson implemented at Telstra in the mobile multimedia services area to troubleshoot problems and investigate incidents. The solution was initially implemented to provide coverage of the 2008 Beijhing Olympics. Telstra predicted massive interest for mobile streaming yet demand exceeded all expectations. Splunk helped Ericsson and Telstra quickly pinpoint, manage and address problems. Because application failures and limits were discovered before they cause serious downtime Telstra maintained an uptime above 99.9% during the Olympic Games.



Telstra manages more than 10M users and 50 plus content providers on the Telstra Service Delivery Platform providing multiple mobile portals, content transformation, mobile streaming services and device specific rendering and UI over 2G and 3G networks. The environment consists of 60+ servers (Solaris 9/10, Windows 2003) and many platforms and technologies providing service orchestration, rich media content management, encoding and streaming for terabytes of active content.

Ericcson and Telstra’s challenges before Splunk were numerous including:

• no central view of logs and events resulting in difficult to troubleshoot problems,
• support and operations diverted to log fetching and ad-hoc reporting delaying work on high priority projects,
• no consistent approach to log handling and storage making it difficult to locate, access and archive logs and
• poor visibility of service and transaction flows extending outages.



The Ericsson team chose Splunk to help Telstra gain a holistic view of the environment, troubleshoot outages more quickly, provide users with ad-hoc reporting and control access to logs with by role. They are currently indexing roughly 20GB per day on a dual processor, dual core Xeon GHz server with 16GB of RAM. 30 support people (tier 1 and up) currently Splunk application, server and network logs and events to troubleshoot problems. The team makes extensive use of Splunk tagging to create alerts for future notification of problems reoccurring. Perhaps the most valuable thing Ericsson has done with Splunk is track end to end transactions on the Service Delivery Platform. With one view across all services and transactions to track activities the team can finally provide transaction level alerting and reporting.



Thank you again to Nick and Martin for presenting so well and Monsour, Martin and Sky with DNA who did a fantastic job and are representing Splunk very well down under.



]]> http://blogs.splunk.com/thebaum/2009/08/27/splunk-4-down-under/feed/ http://blogs.splunk.com/thebaum/2009/08/19/splunk-4-lands-in-the-southwest/ http://blogs.splunk.com/thebaum/2009/08/19/splunk-4-lands-in-the-southwest/#comments Thu, 20 Aug 2009 03:29:40 +0000 thebaum http://blogs.splunk.com/thebaum/?p=245 Last week we continued our road show launching Splunk 4 through the Southwestern US in Phoenix, San Diego and Los Angeles.This was our second annual gathering of customers, partners and users and we had more than double the attendees at this year’s Splunk Live events. In the morning we held a three-hour hands on technical workshop. Attendees had the opportunity to install and configure Splunk 4 on their laptops or remote server and get one-on-one assistance from the Splunk team. Afternoon sessions and dinner focused on customer presentations. We’re very grateful to all the presenters who took time out of their busy days to share with everyone how Splunk is transforming their IT environments. I captured some notes from the week and thought I’d share them with you.

Early Warning

In Phoenix we had a packed house at the Sanctuary conference center on the side of Camel Back Mountain. At 109 degrees I decided against hiking up it in the early AM. Dave Bridgeman, Data Security Engineer at Early Warning kept things cool showing the audience how his company’s use of Splunk in their security operations center. Early Warning collaborates with major financial services companies to facilitate fraud detection through shared information and knowledge in cross-institution environments. The company has an interesting history having spun out of First Data and is now primarily owned by Bank of America, BB&T, JPMorgan Chase and Wells Fargo.

Dave is a well rounded IT professional who started as a developer then moved into network and security management. He current leads the data security team for Early Warning. The environment he over sees includes a variety of platforms including AS400s, MP300s, AIX, Solaris, Linux and Windows. He uses a combination of Splunk forwarders and syslog forwarders to collect Java and Cobol application logs and FTP/SFTP networking logs.



The Early Warning Splunk installation is designed to track transactions and users from one bank to the next in cross-institution activities. Transaction ID tracing correlates events across applications and services and Splunk alerts the team when jobs fail so the operations and development teams can securely troubleshoot issues on the fly. And remote accessibility mean no more driving into the office to access locked down servers in the middle of the night. On the security side of things Splunk helps Dave’s team track and monitor known fraudsters and bad user names allowing them to stay vigilant when monitoring external attacks. They also use Splunk to deliver reports for customers, executive committee members and the Security Advisory Committee (with representatives from the founding banks).

Amkor

Henry Grant of Amkor a $2.1B provider of packaging/assembly and testing services for the semiconductor industry also presented an overview of how his Corporate Data Center team uses Splunk. Henry overseas operations for the company’s SAP, PLM, Supply Chain, Hyperion and Oracle systems. Amkor has a heterogeneous environment of Sun Solaris, IBM iSeries, Cisco ASA firewalls, packaged and custom web and J2EE applications and TACAS/Radius accounting and access control technologies. With manufacturing locations in China, Japan, Korea, Taiwan, Singapore and The Philippines and headquarters in Chandler, AZ, the Amkor team is challenged with log and event data overload. GBs of data a day generated at multiple points makes operational troubleshooting and security investigations extremely complex.



SOX Compliance

Proving SOX compliance has traditionally been handled by writing and maintaining scripts to collect and report on errors, access controls and log access activities. It was impossible to segregate duties given the lack of access control to the logs and events themselves. Splunk has taken the place of the awkward script writing and maintenance to collect iSeries, Unix and application events and logs and provide automated schedule reports. The team is now expanding the Splunk footprint to handle network and Oracle logs as well.

Application and System Monitoring

Like most enterprise IT shops, Amkor has figured out that traditional point monitoring tools aren’t enough as they have a hard time scaling to all the modern day technologies, require intrusive agents and only work for known events but don’t handle anomalies and unknowns. Too many issues end up being reported by end users themselves rather than the monitoring systems. With Splunk Henry’s team detects event anomalies in real time and has dramatically cut their response time by hours per incident.

Tools for the Help Desk

Sometimes it’s the simple things that can cut your response time, escalations and IT budget. The Amkor team noticed a lot of calls and emails regarding VPN set-up and access across the company. With Splunk level 1 help desk agents are now able to resolve most of the VPN issues without creating an escalation. Henry’s team built a VPN dashboard driven by a series of searches and reports that gives entry level help desk personnel the insight they need to troubleshoot problems right away.

Henry’s Splunk Tips

The best part of Henry’s overview were the tips for a successful Splunk implementation. I’ve included the list here in hopes that these may help you as well.

• Provide training that caters to each group’s need.
• Utilize the deployment Server.
• Develop a Common Information Model.
• Update and change as needed.
• Use Tagging to Normalize Data.
• Monitor Scheduled Compliance Reports by using the Audit Logs.
• Splunk into your processes where possible.
• Setup Test/Dev Environment and a Test/Dev Index .

Intuit Consumer Group

The Intuit team of Jeff Ludwig, Chief Architect and Larry Raab, Architect of the Consumer Group joined us to share how use Splunk in production support operations. Jeff leads the Consumer Group’s Connected Services Development for electronic and print tax and payroll filings for TurboTax, ProSeries, Lacerte and QuickBooks. Larry speciali a large-scale, highly available application and systems architect responsible for the consumer group applications and infrastructure.



While the original use for Splunk at Intuit was application management, Jeff and Larry covered three additional ways they have applied Splunk including reliable monitoring, improving user experience and large-scale reporting for compliance and business intelligence.

Application Management

Inuit’s Consumer Group problem is very common. Several services, dozens of machines per service, dozens of log files per machine. Tracking down error logs took hours and correlation across logs and services was nearly impossible. With Splunk the team finds answers in minutes, keeps developers off of production machines and can now correlate across the entire organization and environment – something that is providing them with incredible new insights.

Reliable Monitoring

Jeff and Larry summed up their legacy monitoring systems in this way, “Monitoring tells us WHAT, but Splunk tells us WHY.”

The Intuit Consumer Group team uses lots of other monitoring and alerting tools for networking, servers and applications, but Splunk tends to be more reliable and is the most powerful in terms of features and speed. But the biggest advantage Jeff and Larry see to integrating Splunk with their current monitoring systems is that they can create ad-hoc alerts with Splunk – getting smarter about their environment on the fly.



Improving User Experience

For Intuit’s Consumer Group, when it comes to tax and payroll offerings every transaction completion is critical. But, each transaction goes through several services and many different technologies. Splunk consolidates disparate pieces of the transaction environment so the team knows when something goes wrong and how to fix it.

As Jeff points out, “with Splunk we’re get more intelligent about our users behavior so we can offer them a smarter and better experience.

Large-Scale Reporting

Consolidating Intuit’s Consumer Group’s messages, events and logs has finally make reporting easier and faster for

• Internal data and security audits.
• Financial audits.
• Operational metrics and statistics to plan future deployments and developments.

Thursday we headed up the 405 from San Diego to LA for the last of our Southwest tour. The W Hotel in Westwood was once again the location for our second annual Splunk Live LA. It was a lively scene around the hotel which is just blocks from the Federal building where a police chase ended up in day of traffic snarls and helicopters hovering noisily overhead all day.

Fortunately we had Jon Hart, Manger of Production Engineering at Edmunds and Jeremy Custenborder Senior Performance Architect at MySpace to share how they have deployed and are using Splunk.

MySpace

We were fortunate enough to have Jeremy Custenborder, a Splunk fan and Senior Performance Architect at MySpace drop by to share his experiences identifying and troubleshooting performance issues with Splunk. Jeremy is responsible for performance management across multiple datacenters and thousands of database, web, indicator, index and cache servers and switches, routers and load balancers for MySpace.com.

Lots of MySpace friends generate gigabits of traffic at a time and Jeremy makes serious use of Splunk to keep on top of overall site performance. Jermey says, “Unstructured data rocks!” I happen to agree with him. His advice is to get the data into Splunk, then figure out what to do with it. His Splunk installation includes four indexers per datacenter on a 1 GB network with Raid 1+0 volumes; four cold storage servers on a 1 GB network with Raid 6 volumes and two distributed search servers. Data gets into Splunk in a variety of ways Unix servers use syslog, Windows servers use a custom MySpace agent, .Net applications make use of a Splunk log4net appender Jeremy wrote and has published for others to use as well. The Splunk log4net appender provides both UDP and TCP based transport with failure detection and dynamic configuration via DNS. Why didn’t I think of that? DNS makes total sense for forwarder configuration. You can download the Splunk log4net appender. It is available for use under an MIT license.

Today Jeremy has Splunk performing real time alerting of error data, searches for patterns of suspicious behavior and uses data from Splunk to recreate error in development environments. He plans to start building custom dashboard for development with data specific to each development team and is busy integrating the MySpace performance monitoring system with Splunk to get early detection of new trends and provide fast right click investigation from the performance console.

Edmunds.com

Edmunds has been using Splunk for almost two years now primarily in fraud and security operations. The company is a incredible resource for automotive consumers and enthusiasts. Jon is a self professed Security Ninja and SysAdmin who enjoys racing cars and mountain bikes when he’s not Splunking security incidents. Data comes into Splunk via syslog, a custom agent for windows event logs and .Net application data via a custom log4net appender Jeremy wrote and has published.



Edmunds has more than a thousand devices and servers powering their business with many different logging mechanisms and locations.

Like many enterprises they previously built their own log analysis tools but have replaced those efforts with Splunk. In Jon’s words, “we’ve got better things to be doing around here!”

Edmunds Splunk environment consists of

• 11x 8-core, 64-bit, 16-32G RAM, 300G 15k RPM local disk, 2T NFS (3.4)
• 6 indexers, 2 Splunkweb (1 corporate, 1 production)
• ~60-70G/day, increasing to ~100+G/day soon
• NFS, syslog-ng, Splunk forwarders
• Apache, WebLogic, F5, Oracle, Web Crossing, a metric ton of syslog
• 9 sources, 6 sourcetypes, ~1000 hosts
• distribute search (Splunkweb, CLI) across all indexers
• Centralized Splunk management FTW
• 10 classes outside of per-machine classes
• Multi-membership
• LDAP + AD integration, per-group authorization and

So what is Jon and Edmunds doing with this set-up?

Real-time Alerting and Historical Trending

Edmunds uses Splunk to monitor the good, the bad and the ugly. Good includes traffic trends are tracked and reported on to ensure revenue and analyze trends. Bad consists of port scans, aggressive spidering by search engines and other bots and device failures. And ugly is of course anything that disrupts revenue and Edmunds money making IT look bad.

Developers, engineers, admins, analysts and even managers have visibility into everything. For every application, there are easy Splunk forms for things like errors by environment , host or time including cross-application (think web tier <-> app tier correlation).

For everything that logs data, Edmunds appends a few simple pieces of data that makes everyone’s job a lot easier. I’ve never seen an organization so organized with their logs and events!

• Environment (PROD, TEST, QA, DEV, etc)
• Tier (App, Web, DB, Admin, etc) and
• Normalized source name (“apache” instead of /var/log/httpd/…)



Using this simple organization and a few Splunk search commands, Edmunds drives a series of daily and weekly trends like daily, weekly “Top X” error reports for Web and Application tiers. These trends can also can an eye on the complete build process for monitoring of error diffs between data and build numbers allowing Edmunds to catch error before production code rolls. Developers, not administrators can now monitor and diagnose errors during the development process more effectively. Recently this type of diagnosis and trending has been used to even prioritize development tasks. For example, when someone complained that a particular feature didn’t work with a particular version of Microsoft Internet Explorer, the developer in charge used Splunk to become the voice of reason, discovering the issue impacted only 0.06% of traffic to Edmund’s web sites.



Security

Edmunds has taken a similar approach to simply organizing their Security logs and events by normalizing data from Cisco devices, Netscreens, Sourcefire and Access Control Systems. Normalized fields include src_ip, dst_ip, src_port, dst_port, and protocol. So searches like startdaysago=1 src_ip=1.2.3.4 dst_port=80 will work regardless of log format. Now Jon can easily answer the question of “Who done it?” Without a single source for all security data and cross-device correlation that was previously this use to take a long time and often be impossible.



Before and After

Jon offered this comparison in a example of life before and after Splunk. Edmunds makes heavy use of HTTP logs for all kinds of work. Recently an HTTP log from 6/5/2009 (7G compressed, 60G uncompressed, 115M events) was used with a goal to find the top 10 referrers generating 404 (not found) errors. Before Splunk he’d Gzip/grep/awk/sort in about 7 minutes time. With Splunk he can index in Splunk, search, sort in a mere 58 seconds. Summary indexing in Splunk reduces that to 13 seconds. And this is all on Splunk 3.10. When Jon migrates to Splunk 4 he will be 5 to 10 times faster still.

Summary indexing is a great way to calculate ongoing stats in Splunk and Edmunds makes use of it not just for referrers but for status, method, URI, and UserAgent. Then they combine summary indexes for status, method, URI and referrer across WebLogic, Oracle, Tomcat and Apache to baseline different types of transactions and monitor anomalies.

The Bottom Line

Even though Jon is highly technical, he has been incredibly effective at translating the benefits Splunk brings to Edmunds in business terms. He’s learned this is the only way IT gets to make new investments. He justified the purchase of Splunk by demonstrating it has drastically reduced MTTR for revenue impacting incidents and helped ensure a steady flow of online ad revenue from the four Edmunds Web sites. But the IT and Security teams at Edmunds know there are a number of other advantages. The continuous improvement through automated error reporting and trending, elimination of the “log god” bottleneck, much more productive cross-team debugging and investigations and being able to satisfy that “I wonder if . . .” curiosity in the every day course of doing their jobs are all make their jobs a lot easier to do.

What’s Next at Edmunds?

The Splunk deployment continues to move forward at Edmunds. On Jon’s list of improvements for the next several months are

• Dedicated summary indexers.
• Redundancy.
• Longer retention periods.
• Double indexing volume by 2010 (more RAM, more storage) .
• Windows event log.
• Splunk 4.0 migration.
]]> http://blogs.splunk.com/thebaum/2009/08/19/splunk-4-lands-in-the-southwest/feed/ http://blogs.splunk.com/thebaum/2009/08/01/splunk-live-london-awesum/ http://blogs.splunk.com/thebaum/2009/08/01/splunk-live-london-awesum/#comments Sat, 01 Aug 2009 18:20:24 +0000 thebaum http://blogs.splunk.com/thebaum/?p=225

I’m finally getting my head above water after a tireless run up to and hectic week launching Splunk 4. The highlight of the launch for me was Splunk Live London. IMHO Splunk Live London 2009 was unrivaled as the most outstanding Splunk event yet.
We came up with this idea of getting local customers together as a way to launch Splunk 2 in June 2007. Five of us Splunkers sprinted between eight different cities in two weeks to share what was new and encourage users to exchange stories of how searching their data centers was changing life for the better. Its an exhausting way to launch a new product, but it worked so well we’ve integrated Splunk Live events into the mainstream way we do business and interact with our community. I’ve long since lost count of the number of Splunk Lives we’ve conducted all over the world including places like Cape Town, Johannesburg, Beijing, Tokyo, Singapore, Bangkok, Sao Paulo and yes once again in London.





This year’s London Splunk Live was really special. The event occurred during our launch of Splunk 4 and surpassed our expectations as the largest event we’ve ever held. More than 100 customers and users attended at the Cumberland Hotel and their swank conference facility, complete with a business canteen like breakfast experience, near Marble Arch in West London.

But the dominant reason to attend any Splunk Live are the presentations and round tables with forward thinking IT professionals who are using Splunk to transform the way they manage IT. This year we were very fortunate to have three Splunk customers who took time out of their busy schedules to come to London and share their experiences with us.

Accenture - Alexander Strobl, Technical Consultant

Alexander has been a visionary inside Accenture bringing the power of IT Search to enterprise clients in Germany where he works for Accenture as a Technical Consultant in the Data Center Technology and Opeations team. Alexander is responsible for analysis, design, roll out of Splunk. His most recent Splunk project was with a large worldwide services company with more than 50,000 employees on three continents operating mail order, distribution, e-commerce and over-the-counter-retail trade. Accenture implemented Splunk to transform the management of several technologies including Linux, virtualization and large-scale storage systems.

The project was part of an IT project to reduce the time to triage problems and improve quality of service. Challenges were:

• no centralized access to logs and events,
• critical IT data was stored on local file systems which were copied to central storage only once a day,
• manual processes to locate errors,
• no correlation between events on different services/servers and
• development time was spend building workarounds rather than working on revenue generating applications.

All of this resulted in complex and time consuming analysis and end the end long MTTR.



The Accenture Splunk installation is currently indexing ~50GB/day including custom application files and events from 10+ integrated business critical applications and services. There are two Splunk indexes; one for testing and one for production environments and the team has established interfaces between Splunk and several other legacy data center tools.

Telenor - Henrik Strøm, Security Architect

Telenor is Norway’s largest ISP, Mobile Operator and Telco. Its one of the largest mobile operators in the world, with 160+ million customers and was founded in 1855 - 154 years ago. The company has 13.000 employees in Norway and 26.000 abroad. Telenor has been rolling Splunk out for centralized log collection and management using Syslog to forward data where it is already in place and using Splunk as a forwarder for new systems and systems with complex multi-line and/or XML structures Syslog can’t handle. Sources of data handles by Splunk include:

• application logs (Web, Email, IPTV)
• data center logs (server, network, storage and firewall)
• IP backbone logs

Use cases include what Henrik refers to as digging, dashboards baselines, alerting and reporting. One of the best “digging” examples Henrik mentioned was identifying Unix Kernel Errors over the last 30 days. This kind of information routinely went unnoticed prior to Splunk’s arrival.



Another powerful use case explained by Henrik was how to baseline what is normal in your environment. For example, how many errors do you have on average for a particular type of device (routers, servers, specific applications, etc). Splunk was used to baseline normal Linux kernel behavior and found roughly 20 kernel errors per Linux running instance every 15 minutes.



The base line then allows the team to schedule simple searches to look for deviation from the baseline and send out alerts before downtime occurs from these hidden sways in behavior. In one case Splunk found thousands of errors occurring on a specific type of device, where the normal baseline was around 20!



The Telenor team also uses Splunk to identify and report on security situations that may impact their customer facing network and services. Because they are able to easily compose dashboards showing for example which Web servers are under attack and who is attacking them all in one place, the team saves Telenor from potential downtime, performance degradation or theft of data due to attacks they’ve not seen before and are missed by existing security policies and technologies.



Vodafone - Paulo de Carvalho, Network Services Manager

Paulo de Carvalho has been using Splunk at Vodafone for almost two years now. His presentation titled “Freeing Information from Organizational Silos” lifted the idea of leveraging logs and IT data out of the realm of just system administration into a thirst for higher level intelligence that crosses not only IT but also business functions. Paulo started by describing the current service oriented architecture (SOA) at Vodafone and how attempts to objectize and re-use capabilities creates incredible complexity among the services, technologies, processes, tools and people.



Using Dennis Leeden’s Sense-Making Model, as a blueprint for the raising the intellect of IT and business consumers of IT services, Paulo has proven how achieving a level of “knowledge” versus just “data” management can significantly impact the performance of an IT organization and the services they provide. He went on in detail describing how Vodafone has broken down the segregation of duties along business process, technology services and business units and determined what knowledge is essential and can be provided from the active running IT systems regarding their behavior, performance, configuration, dependencies etc. His team has defined data inputs, searches, reports and dashboards for the most important intersections of processes, services and technologies using Splunk. The impact on the performance of IT and the quality of services to consumers has been dramatic.



IT knowledge management at Vodafone has significantly improved the quality of life for IT people and customers too. If you’ve ever been frustrated dealing with a customer service agent at a mobile phone company, an airline or a government agency you understand a little information can go a long, long way to improving our ability as humans not to use technology as an excuse for poor customer service, but to actually deliver the type of customer service customers appreciate.



If you’ve attended one of our Splunk Lives you know why I’m so passionate about them. If you haven’t attended and think you might be interested, check out our events page for more information about where we’ll be when.

Happy Splunking!

]]> http://blogs.splunk.com/thebaum/2009/08/01/splunk-live-london-awesum/feed/ http://blogs.splunk.com/thebaum/2009/07/21/if-splunk-was-an-animal-what-would-it-be/ http://blogs.splunk.com/thebaum/2009/07/21/if-splunk-was-an-animal-what-would-it-be/#comments Wed, 22 Jul 2009 00:42:39 +0000 thebaum http://blogs.splunk.com/thebaum/?p=224 Splunk 4 is out of the bag and the Splunk community and our customers are kicking the tires. I even saw several executives from other log management, SIEM and system management vendors registered and attended our world-wide webcast with a thousand attendees. And Twitter is all abuzz with questions, answers and some ass kicking. Yes Splunk 4 kicks ass. It is 2x faster on indexing and up to 10x faster searching. We have a fantastic new App framework where you can build custom views, dashboards and work flows and there are countless numbers of other great improvements and new features. But sometimes we don’t get it completely right and you all let us know.

But back to my question, if Splunk was an animal what kind of animal would it be?

“Odd thing animals. All dogs look up to you. All cats look down to you. Only a pig looks at you as an equal.”

- Winston Churchill

I read that quote today at the birth place of Winston Churchill and it reminded me that Splunk is like a pig. We’ve always looks our users and customers straight in the eye with the good and the not so good. This has always been the transparent way we conduct business. So keep the feedback coming - the praise and the criticism.

One of the areas that I’m especially interested in hearing about is our new App focus. We are in the very early stages of creating Splunk Apps and making them available to the Splunk community. Some are free Apps and some are premium Apps. The free apps are available for immediate download. The premium Apps you need to talk with us about so we can work with you on an installation. At some point we plan to have trial versions of the premium Apps available for download too.

The free Apps include things like

• Splunk for Unix and Linux
• Splunk for Windows
• Splunk for Blue Coat
• Splunk for Cisco
• Splunk for use with F5 Networks



You can easily download the App .spl file, drop it into your splunk/etc/apps directory and check it out. More easily you can download and launch the Apps right from your Splunk Launcher screen (which is an App too). We’re working on fully documenting all these Apps so if you need help now feel free to contact us via support@splunk.com. You can also select “Send Feedback…” on the first menu of the App to contact the specific App team directly via email. We’re especially interested in what doesn’t work, where you get stuck and what else you’d like to see. Several of these Apps are still beta versions so feedback sooner rather than later is much appreciated.

Happy Splunk4ing!

]]> http://blogs.splunk.com/thebaum/2009/07/21/if-splunk-was-an-animal-what-would-it-be/feed/ http://blogs.splunk.com/thebaum/2009/06/18/internet-censorship-run-wild/ http://blogs.splunk.com/thebaum/2009/06/18/internet-censorship-run-wild/#comments Thu, 18 Jun 2009 14:13:46 +0000 thebaum http://blogs.splunk.com/thebaum/?p=219 The past couple of days I’ve been visiting China meeting with some of our technology and channel partners. It just so happens I was present in Beijing for the 20th anniversary of the 1989 Tiananmen Square Events. Yes it really did happen despite what the Chinese government says. Speaking on Saturday at the F5 APAC Sales Kickoff I found myself staying over the weekend with Sunday off to roam around Beijing like a tourist, something I rarely get a chance to do on business trips. It is amazing to me to see how the Chinese and Taiwanese work on Saturdays. In the US we rarely see that. Europeans chastise Americans for working too hard but I guess they should really see the work ethic in Asia and then we’d look more normal.



Watching the 2008 Beijing Olympics last summer things there certainly seemed more normal than 20 years ago, but being there in person with all the festivities gone things seemed really strange to me. It is very difficult to describe. Maybe I was jaded by all the newspapers I’d read on the way to Beijing. On a nice long 13 hour flight from Washington DC with plenty of reading material I consumed James Kynge’s piece in the Financial Times questioning whether the Western media really understood why the student demonstrators were protesting. He went on ascribing the word “democracy” with the student motivations and questioning whether we or they really knew what it meant despite the fact that he spells out their desires in plan old English which sounds like democracy to me.

“Almost everything fell within its scope: campaigns against corruption, nepotism, inflation, police brutality, bureaucracy, official privilege, media censorship, human rights abuses, cramped student dormitories and the smothering of democratic urges. But to say the demonstrations were to “demand democracy” is an oversimplification.”
James Kynge, Financial Times

It’s almost impossible to describe the strange feeling I got while walking through Tiananmen Square observing the soldiers and the huge portrait of General Mao that dominates the landscape. Maybe part of it was due to the increased tension of the anniversary. Maybe not. Tiananmen has come to symbolize the unspoken and largely unrecognized tension between the economic progress driving modern China and the old fashion communist government still ruling there. The Chinese seem to have a foot in both camps. The eeriness I felt came not only from my surroundings and an understanding of the principles they stood for but also from the reaction of my Chinese and Taiwanese friends. Their usually jubilant outgoing personalities were completely subdued in the square. Was a sign of respect and mourning that drove their thoughts? Perhaps to some extent. But in quiet whispers and conversations out of the ear shot of any “green” uniformed soldiers (versus the “blue uniformed” security guards they confessed to being actually scared to speak for fear of someone or something listening. Challenging them I said, “surely you must be joking.” But it was no joke. Only when we crossed the street into the forbidden city did their usual personalities return.

Of course this began a prolonged conversation over the next 24 hours as we visited the great wall, a new Beijing restaurant and departed through the impressive new Beijing airport. I kept asking and trying to understand. How can a country of so many people be controlled by the minds of so few? What are the real limitations to speak out? And what effect will economic progress have on the political future of China? There was no shortage of stories supporting the fact that the government still does take a very heavy hand to those who disagree. But rather than discuss it, everyday Beijing seems to sweep the event of 20 years ago under the rug. As one of my Chinese friends said, “everyone is embarrassed and we just pretend it never happened.”

At the same time I was traveling through out China, the articles started pouring in about Beijing’s efforts to step up Internet and IT censorship. Upon reading the perspectives pouring in about “Green Dam” I was reminded of the impact the technology industry is having on the whole situation. It was bad enough I couldn’t get to sites like Twitter and Youtube form my hotel room. Now the Chinese government is requiring every PC sold in the country starting July 1st has to have special software blocking all sorts of things. The move is being presented as an attempt to protect children from online pornography but is obviously one more attempt by Beijing take its censorship to a new level. China currently has the world’s most sophisticated and multi-layered system of Internet censorship. Objectionable content on domestic Web sites is deleted or prevented from being published, and access to a large number of overseas Web sites is blocked or “filtered.” Decisions about what to censor are based on the Chinese government’s attempts to control the minds of 1.2B Chinese. There is no transparency or accountability, no public consultation in developing block lists or censorship criteria, and no way to appeal the blockage or removal of Web content.

In a notice to PC makers, the Ministry of Industry and Information Technology said all PCs shipped in China needed to offer Green Dam/Youth Escort, identified as a “green internet filtering software”, either pre-installed or as part of basic software packages. In May 2008, the government picked Jinhui Technology and Dazheng Language Technology, two Chinese software companies to develop the software, according to a contract award notice from the MIIT. While these companies claim their software is only being used to block sites although last year, researchers discovered that a Chinese version of Skype contained the ability to block politically sensitive words in instant messaging chats, and to keep a record of the use of such words.

While there is obviously a legitimate role for filtering software, we’re starting to see governments take this way too far. Green Dam is only one example of a global trend. Internet censorship is expanding rapidly and now includes a growing number of democracies. Legislators are under growing pressure from family groups to “do something” in the face of all the threats sloshing around the Internet, and the risk of overstepping is very high. In China’s case it’s an open door to abuse power in the attempt to prove the legitimacy of an ailing legacy.

]]> http://blogs.splunk.com/thebaum/2009/06/18/internet-censorship-run-wild/feed/ http://blogs.splunk.com/thebaum/2009/04/23/conficker-is-proof-we-need-to-log-broadly-and-analyze-deeply/ http://blogs.splunk.com/thebaum/2009/04/23/conficker-is-proof-we-need-to-log-broadly-and-analyze-deeply/#comments Fri, 24 Apr 2009 05:04:51 +0000 thebaum http://blogs.splunk.com/thebaum/?p=218 At RSA this week it’s easy to got lost in the menagerie of security technologies to conquer malware proliferation, stomp out spam and protect virtualized and cloud computing environments. But the most recent statistics show we are still losing the war on cybercrime. Symantec’s latest Internet Security Threat Report sited 1,656,227 malicious-code threats last year and 75,158 new active bot-infected computers per day. And yes the United States is still the most frequently targeted by denial-of-service attacks accounting for 51% worldwide and the top country for underground economy servers advertising stolen credit cards accounting for 67% of all activity worldwide.

Why are we losing so badly? Not surprisingly, there was a lot of talk at RSA about the Conficker worm. Some of the chatter points to reasons why the security industry is falling behind. At first glance, the Conficker worm looks harmless. So far there are not too many significant reports of infected machines and hijacked data,
but it may be too early to feel so smug about it. The worm’s real danger is its demonstrated ability to evade the expensive IDS technology enterprises have put into place and rely on today. Estimates are that 90% of the enterprise IDS implementations have failed to detect the worm’s presence and create some kind of actionable alert. How can this be?

Conficker properties are simple but different from the typical threat. First Conficker affected systems outside of IDS coverage like USB keys and mobile user laptops. So if you’re looking for attacks from outside your network only, you won’t see it. It’s a “walk-in virus”. Second it isn’t greedy like Code Red and other viruses of late. The Conficker worm has built-in sleep cycles. So where a typical worm might scan 1,000 or 10,000 IPs a minute, Conficker was happy to scan maybe say 100 and evade the baseline trip wires. Third Conficker is very selective with its payload delivery. It only delivers when it sees a vulnerability. All this helps Conficker evade IDS systems that want to witness the crime. But Conficker is the perfect crime in that it goes undetected. With no payload delivered and seemingly fewer IPs scanned there is no grossly abnormal behavior to witness. The evidence is circumstantial.

At a lunch on Wednesday, Tom Le of BT gave a good overview of how BT Managed Security Services detected Conficker for their customers. It was one of the first times I’ve really been sold on a managed security service beyond the value of cost and convenience.

First, as Tom explained it, they started by assuming IDS would miss the attack. They didn’t assume a payload had to be delivered and didn’t assume that large number of scans were needed to indicate the presence of an intruder. Instead of depending on IDS, BT uses logs and events to baseline the natural behavior of even netbios triggered scans (which Conficker happened to use) and was able to alert on small changes in scans that would be missed if you were only looking at things like netflow. As it turns out most firewalls blocked the netbios scans going out so again most customers didn’t even know they had the Conficker worm present.

Second Tom and his team assumed some type of command and control activity associated with Conficker. They followed the money watching for things like confikur trying to phone home in different ways. By having a broad set of logs and events from switches, routers, applications and IDS they were able to look for outlying behaviors like DNS lookups to obscure locations not typically seen in customer networks and aggregate this information across customers to identify common abnormalities. Tom estimates that BT sees roughly five billion messages a week across their customer base. That’s a lot of messages.

After listening to all the chatter about Conficker and walking the show floor, it gets easier to understand how criminals continue to evade the security infrastructure enterprises put in place. There are just too many ways in which breaches can occur and there is just too much data scattered about to collect and correlate in order to find the anomalies. So the security industry continues down the path of specific solutions to specific vulnerabilities and criminals continue to create new threats that evade the industry’s point approaches. I say the industry as a whole needs to move to more of an adaptable and flexible approach that can apply security to what ever threats arise, when they appear.

The best real world detectives are able to piece together seemingly circumstantial evidence and sift out the clues that lead to catching criminals. But every time it’s different. Perhaps we need to take the same approach in order to obtain more adaptable security solutions. Assume every time it’s different not the same.

Logging broadly and analyzing deeply is one of the best defenses. Without a broad swath of data you won’t have the pieces of the puzzle to put together at the moment you need to solve the crime.

Few criminals are caught in the act.

]]> http://blogs.splunk.com/thebaum/2009/04/23/conficker-is-proof-we-need-to-log-broadly-and-analyze-deeply/feed/ http://blogs.splunk.com/thebaum/2009/02/03/splunk-powered-associates/ http://blogs.splunk.com/thebaum/2009/02/03/splunk-powered-associates/#comments Tue, 03 Feb 2009 22:52:32 +0000 thebaum http://blogs.splunk.com/thebaum/?p=217
Well if you ever wanted to integrate Splunk into your own product or service, free is now really, well … free. We’ve always had a free Splunk license for end users. But now we have the same for software, hardware and service provider partners. Now as a Splunk Powered Associate you can distribute Splunk with the free license key as part of your offering. You can also link to the Splunk free license download and earn referral credits if the download leads to a purchase. Pretty cool heh? Now the free license is still limited to the 500MB daily uncompressed indexing volume but hey that’s a lot of data for free.

A few of our Splunk Powered partners have picked up on the real potential here. F5 Networks, for example, has created a Splunk App that pre packages searches, alerts, reports and dashboards for F5’s ASM and FirePass products. Now F5 customers get real-time search, alerting, reporting and analytics for free with Splunk for use with F5 Networks. Support for F5 LTM and BIG IP is coming soon.

And the folks over at RightScale are taking Splunk into the clouds. RightScale is a great cloud computing management platform that let’s you control your cloud resources across several different providers from one interface. We use RightScale at Splunk to control our demo instances on Amazon EC2/S3. Each demo instance consists of one or more servers running in the cloud that recreate a live IT environment like a J2EE-based E-commerce application, a converged network or a rack of Microsoft Windows Servers. It’s important that we are able to scale these instances up and down dynamically and RightScale comes to the rescue. The integration of Splunk and RightScale gives cloud us the IT control and visibility we need.

Every piece of software, hardware and service on the planet generates IT data. And now you can bring Splunk to your community by integrating it into your solution at no cost to you, your channel or your customers. To join the Splunk Powered Associate program just Sign-up to be a Splunk Powered partner and we’ll take it from there.

Happy Splunking!

]]> http://blogs.splunk.com/thebaum/2009/02/03/splunk-powered-associates/feed/ http://blogs.splunk.com/thebaum/2008/12/05/splunk-live-san-francisco-its-about-time/ http://blogs.splunk.com/thebaum/2008/12/05/splunk-live-san-francisco-its-about-time/#comments Fri, 05 Dec 2008 16:13:39 +0000 thebaum http://blogs.splunk.com/thebaum/?p=209

Last night we hosted more than 100 people at our first ever Splunk Live in San Francisco. It was about time. In May 2007 we started our first series of Splunk Live events. We’ve traveled all around the world from Santa Clara, Los Angeles, Phoenix, San Diego, Dallas, Chicago, New York, Washington DC, Atlanta, London, Zurich, Singapore, Taipei, Shanghai, Bejing, Bangkok and Hong Kong. But never have we had an event in our own backyard. Congratulations to Steve Sommer and our Marketing Team for pulling it off.

The event took place in our new offices at 2nd and Brannan Street.

Little known fact that for the first two years at Splunk we actually never had an office of our own but squatted in the offices of venture capitalists and other start-up companies like Six Apart. Having a conference room called “BIG” where we can actually fit more than 100 people still takes some getting use to.

The best part of course to every Splunk Live are the customer presentations. Last night we were honored to have three local customers show everyone how they are using IT Search.

• Mashery, The leading provider of API management services enabling companies to easily leverage web services as a distribution channel, discussed how they use Splunk to power self-service reporting for their customers on activity within their hosted, cloud-based services.
• Lawrence Livermore National Labs LLNL, a US Dept of Energy national lab talked about their Splunk deployments in multiple groups and data centers addressing a wide range of needs, from application availability to meeting FISMA security regulations. They drive a range of initiatives from high performance computing to nuclear weapons development to running particle accelerators.
• Visa International- The world’s largest retail electronic payments network, and one of the most recognized global financial services brands, will share how they use Splunk for network security monitoring and incident response.

Stay tuned to our events page for more upcoming Splunk Live events next year. We plan to visit several cities each quarter and will likely be in your neighborhood at some point in the near future.









]]> http://blogs.splunk.com/thebaum/2008/12/05/splunk-live-san-francisco-its-about-time/feed/ http://blogs.splunk.com/thebaum/2008/11/06/human-and-machine-launguage-mashups-at-splunk-live-zurich-switzerland/ http://blogs.splunk.com/thebaum/2008/11/06/human-and-machine-launguage-mashups-at-splunk-live-zurich-switzerland/#comments Fri, 07 Nov 2008 03:54:03 +0000 thebaum http://blogs.splunk.com/thebaum/?p=201 At Splunk Live in Zurich this week an interesting discussion erupted about human and machine languages. Before I continue with the story, I want to thank everyone that attended the event. Despite the fact that Raffy Marty is a resident celebrity, this was our first formal customer and partner event in Switzerland. We had more than 50 people attend for several hours to talk about Splunk and data center management challenges. The event was co-hosted by T-Systems.

Thank you Meno Schnapauff for your great presentation on how T-Systems and the Swiss National Railway are using Splunk!

Other attendees included folks from Swisscom, Unicom Consulting, Rothschild Bank, Genossenschaft Migros, LeShop, Netcetera, Cablecom GmbH, TBK-Patent Munich, On Line Video 46, Skyguide, PostFinance and the Univestity of Fribourg. Brian Haynes, Tim Thorpe, Julie Duncan and Hash Basu-Choudhuri from our London office participated too.



Now part of the reason I mention all these names (in addition to thanking folks) is to the point of this post. In the room we had an American (me), several native English speakers from different areas of England, Swiss German speakers from Switzerland and German speakers from Germany. What I noticed is how two people think they speak the same language but can’t always understand each other. It turns out there are a lot of American (some West Coast) colloquialisms I use that my “queens English” counterparts don’t understand. And of course most of the time I try to make a joke the Swiss and Germans just look at me like I’m from outer space even though if you asked them they’d say they speak fluent English. During the event the Swiss Germans had trouble understanding the Germans and the Germans had trouble understanding the Swiss Germans. The folks from the UK who spoke German didn’t understand either the Swiss German or the German German although they all claim to speak German.



What does all this have to do with IT you ask? Well it turns out that mashing up languages and attempting to understand each other even though we don’t speak exactly the same language is one of the biggest problems we have in trying to understand our IT systems as well.

“One of the questions posed at the event was how can I modify my system and application logging to some standard in order to follow what my systems are doing? Do we need a logging standard?”

I have long been telling people that logging standards are a waste of time. IBM’s Common Base Events (CBE) has been around for decades and has very little traction in the real world. Data Center Mark-up Language (DCML) was pushed by Opsware and lots of smart people. It got nowhere. Logs exist. Instrumentation exists. Our IT systems already have tremendous amounts of data. Trying to retrofit that data to some standard is impossible. Attempting to organize a multi-vendor logging standard will never happen. Getting developers to log consistently sounds great but I’ve never seen it done before.

What we need is a mashup of machine languages and logging formats. That’s exactly what IT Search is!

Humans need to stop thinking about how we can format data to make it easier for machines to work with it. There is too much data. The real value is being about to work with massive amounts of data without any human intervention. This is exactly what Google does for the web. Sure you can reformat your HTML to get better search results. But even if you do nothing Google will index your site. You don’t even have to tell Google to do it!

I’m going to start sharing more of our experiences helping people see the connections that already exist in their logging data. While the connections are not always obvious to the naked eye and human linear thinking, machines are great at teasing out non-obvious relationships. This is perhaps the most compelling thing we work on at Splunk and continue to push the bleeding edge of what’s possible.

]]> http://blogs.splunk.com/thebaum/2008/11/06/human-and-machine-launguage-mashups-at-splunk-live-zurich-switzerland/feed/ http://blogs.splunk.com/thebaum/2008/10/30/splunk-voted-fastest-growing-company-in-silicon-valley/ http://blogs.splunk.com/thebaum/2008/10/30/splunk-voted-fastest-growing-company-in-silicon-valley/#comments Fri, 31 Oct 2008 05:59:10 +0000 thebaum http://blogs.splunk.com/thebaum/?p=193

I’ve just returned from the Deloitte Technology Fast 50 awards dinner where Splunk was selected as the fastest growing company in Silicon Valley. Delloite, Silicon Valley Bank, Korn Ferry International, Cornish & Carey, Cooley Goward Kronish and adb Insurance Services were the sponsors of this year’s competition and we thank them all for the award.

I was joined at the awards dinner by my two co-founders Erik Swan and Rob Das. What a great ride it has been over the past four and a half years. The time has flown by so quickly and it seems like we still have so much more to do. But it was nice at least for one evening to take a breather and enjoy what we have accomplished.

Since I graduated from college with a degree in computer science I have dreamed of creating a technology and a company that had the potential to achieve what Splunk has. Seems unreal that we are now here living that dream.

The award ceremony was held at the Computer History Museum in MountainView, CA. What a cool place. When the Boston Computer Museum closed in 1999 the museum in Silicon Valley became the keeper of computer technology history. Wandering through the museum I spotted an exhibit on chess software competition and was reminded by one of the long job outputs hanging from the ceiling of my own chess playing Pascal program that performed a pretty good six level look ahead algorithm.

But it was entering the hardware history wing that really sent me down memory lane.

PDP8s, PDP11s, original IBM PC, Osborne, Apple Lisa, Apple IIc, Mac 128k, Compaq luggable, Apple Powerbook 170 and 230 with that cool ejectible enclosure that hooked up all your cables for you. Wow!

I even saw an IBM 5100. Perhaps the most bizarre machine I ever programmed. It has a switch that moves the shared program and memory space from APL to Basic - two worlds that should never co-exist.

When I was at IBM in Boca Raton I wrote an inventory management system on a 5120 the predecessor with a 9 inch screen!

If you’ve never been to the museum you really should go. Take your kids. Show them the progress technology has made during your adult lifetime and let them dream about the next 25 years.

Where else can you sit on the built in sofa of a Cray 1 supercomputer and see a PDP1 still working to play the world’s first video game?

Thanks to all the sponsors for hosting the event and selecting Splunk as the fastest growing company in Silicon Valley!


The Award - Where’s the cash?



Splunk Founders - Erik, Michael, Rob



How Many Can You Remember?



PDP8



PDP11



Cray 1
]]> http://blogs.splunk.com/thebaum/2008/10/30/splunk-voted-fastest-growing-company-in-silicon-valley/feed/ http://blogs.splunk.com/thebaum/2008/10/21/splunk-lab-in-asia-launches-to-develop-new-it-search-apps/ http://blogs.splunk.com/thebaum/2008/10/21/splunk-lab-in-asia-launches-to-develop-new-it-search-apps/#comments Tue, 21 Oct 2008 20:17:28 +0000 thebaum http://blogs.splunk.com/thebaum/?p=169

The last two weeks I’ve been traveling throughout Asia with our new partners at Systex and the Splunk Asia team. In Singapore, Hong Kong, China and Taiwan we met with government agency, high tech manufacturing, insurance, online gaming and managed service provider customers who told us how critical Splunk is to their IT organizations, especially as budgets get even tighter.

Systex is now our master distributor covering Taiwan, China, Hong Kong, Singapore, Thailand and Malaysia. Systex is an amazing company fueled by Taiwanese entrepreneurship, creativity and innovation. The company is part distributor, part reseller, part system integrator and part independent software developer. The 2,900 Systex employees are led by CEO Hilo Chen and COO Frank Lin. Hilo did a stint at Yahoo! Asia before joining Systex as CEO. He is a very friendly, engaging and good nature executive who commands the passion of his team. Frank is detail oriented and intense and he has an ability to focus on what seems to be the impossible and get it done.

I’m not used to people pushing faster than I do, but the Systex team are reminding me what start-up speed is all about.

The Systex system integration and software business is fueled by more than 1,400 engineers with deep domain expertise in financial trading and banking systems, network security, database administration, storage, virtualization, disaster recovery, IT service management, telecommunications OSS/BSS, unified communications, business intelligence and more. This past week we unleashed the creativity of more than 400 of those engineers, product managers, sales personnel and business unit heads. We met at a three day kickoff event for the launch of a joint Splunk Lab designed to come up with new areas to apply IT Search and new Splunk Apps for a variety of use cases.

It is our hope that our joint work together will result in lots of new Apps available for download by Splunk users all over the world.

The event started Thursday with a press conference at the Westin in Taipei. We were joined at the press conference by more than three dozen press covering innovation in Asia. We discussed the design of the partnership, the Splunk Lab and some of the joint customers including Allianz Insurance, IAH Games, and The Malaysian Prime Minister’s Office. Allianz is using Splunk to report on F5 Big IP load balancer activities. IAH is mining their online multi-player game events and logs for insight into user patterns and activities including market basket analysis across different game properties. The Malaysian PM’s office uses Splunk to secure their email messaging system.

The press asked some very good questions about various use cases and our strategy for accelerating activities in Asia with Systex. Richard Tang and Johnny Lin attended the event from Systex as well and provided a great overview of how the Splunk Lab is coming together and what kind of solutions Systex is creating around Splunk. Richard has been very patient with me and has taught me enough Mandarin to completely embarrass myself during my last few visits.

On Friday 260 engineers and product managers attended an all day Splunk Boot Camp at the Systex UCOM training center in downtown Taipei. The day was divided into two three and a half hour sessions. Each session covered using, administering and deploying Splunk. There was a brief section on developing Splunk Apps including building of a network management application.

One of the product managers commented to me at the end of the day, “My mind is broken on Splunk, there is so much you can do with it.”

Saturday’s session was the Splunk Lab kickoff event and creative activity attended by 300 business unit heads, sales people, product managers and field sales engineers. I was amazed. We went from 8:30am to 6:30pm on a Saturday. The level of energy was unlike anything I’d ever experienced before. Taking the long trip back from Taipei by way of Tokyo, I am just in awe at how two organizations half a world a part have so tightly bonded in just six months. I’m very impressed by the Taiwanese work ethic and dedication.

Kord Campbell, Splunk’s Director of Developer/ISV program gave a great talk on developing Splunk Apps to start the working round tables. Each business unit (twelve in all) spent three hours coming up with ideas for Splunk in their unit including what Splunk Apps they were going to create and which customers they were targeting. The areas included

• Financial Trading Platforms
• Banking and ATM Systems
• Database Serivces
• Information and Security
• Business Continuity and Disaster Recovery
• Customer Service
• Data Management & Integration
• Unified Communications
• IT Service Management
• Education & Training

Teams were judged on several factors including creativity, feasibility, significance to current business and target customer profiles.

The winning team didn’t use slides but instead acted out their presentation in a 15 minute skit. It was wild and reminded me of how dysfunctional most IT organizations are today. Not that we needed reminding :-)

The Financial Services Business Unit was judged the winner. This team has developed market trading platform software in a joint venture with Reuters and explored using Splunk with their quotes and trading solutions and for market compliance. The first scenario involved monitoring TAIFEX, TWSE and OTC trades and examine patterns indicating potential fraudulent activities.



The second scenario showed how IT Search can be applied to troubleshooting the electronic system including buy side, sell side, cash position, web interfaces, trading systems and risk management. Actors in the scenario ranged from investors, web infrastructure managers, dealer groups, trading managers, CRM users and back office personnel. The team called their solution “A Lighthouse in the Dark.”



Perhaps the most interesting integration of Splunk though was the mining of data from the web application platform to determine which features users tapped into and which ones they tried once but never went back to. By examining page views for new functions and correlating those with trade volume deltas the team can continuously monitor the revenue effects of application and site changes.



The Splunk Lab launch has us thinking about how to get other people collaborating to build new applications for IT Search. We’re planning to launch a public site soon that will allow domain experts from all over the world to work together and create great Splunk Apps. So we decided to take the elevator to the top floor of Taipei 101, the world’s tallest building to look for more…




Top Floor at Taipei 101

View to the East of Taipei
Press Conference

Frank Lin, COO, Systex

Me

Robert Lau - Splunk & Emy - Systex

Hilo Chen, CEO, Systex

UCOM Technical Training Center
Kord Campbell - Splunk

Splunk Lab Team Competition

Winning financial services App

A little bit of fun
Taipei 101 - World’s Tallest Building ]]> http://blogs.splunk.com/thebaum/2008/10/21/splunk-lab-in-asia-launches-to-develop-new-it-search-apps/feed/ http://blogs.splunk.com/thebaum/2008/10/06/splunking-across-the-pond-welcome-brian-haynes-vp-emea/ http://blogs.splunk.com/thebaum/2008/10/06/splunking-across-the-pond-welcome-brian-haynes-vp-emea/#comments Tue, 07 Oct 2008 07:42:09 +0000 thebaum http://blogs.splunk.com/thebaum/?p=167

It’s kinda a funny story and although it seems so long ago it was just 18 months ago. I was traveling in Europe starting to talk with potential customers who had downloaded and installed Splunk (3.0 variety). My very first meeting was with a guy name Scott Davies VP of E-commerce Trading Platforms at Royal Bank of Scottland in London’s Bishop Gate. I had the opening slide to our presentation up when Scott walked in the room. He was very polite, asked us if we wanted some still or sparkling water and wanted to know how our trip was progressing thus far. Finished with the pleasantries he than quipped, “I love your product, but when are you going to change your name.”




Seems “Splunk” didn’t quite translate all that well in the UK. Although Colin Barker and Steven Arnold didn’t seem to mind. Fast forward to October 2008 and here we are with more than 60 customers in Europe including several major banks, telecommunication providers and large enterprises. And now we have a big shot head of EMEA and an incredible team on the ground in London. Welcome Brian Haynes!

I first met Brian about three months ago at the Berkeley Hotel in London. We hit it off immediately. Brian was incredibly excited about our free download model as he had experienced similar success with companies like Legato that initially followed a simlar model. The difference he said was, “Splunk really believes in fostering a global community of users around its product, something Legato never had.” As our new Vice President Sales for EMEA, Brian will no doubt help us really accelerate our growth in the European market. He joins us at a great time. Last week we attended the IP 08 show and our booth was mobbed with folks anxious to learn how they can Splunk their infrastructures.

As the global economy continues to crumble its amazing to see that we’re able to keep bringing value to customers around the world and grow our user and customer base by helping IT organizations do a lot more with less. The notion of a single universal platform that breaks down the silos between operations, security and compliance will certainly continue to thrive.

]]> http://blogs.splunk.com/thebaum/2008/10/06/splunking-across-the-pond-welcome-brian-haynes-vp-emea/feed/ http://blogs.splunk.com/thebaum/2008/09/19/splunking-vmware-virtualization-at-vmworld/ http://blogs.splunk.com/thebaum/2008/09/19/splunking-vmware-virtualization-at-vmworld/#comments Fri, 19 Sep 2008 15:00:51 +0000 thebaum http://blogs.splunk.com/thebaum/?p=165 This week things were rocking and we were splunking at VMworld. VMware launched their road map for their Virtual Data Center Operating System (VDC-OS). VDC-OS is VMware’s vision to aggregate virtualized servers, storage and network resources into a common platform that manages resources for guest operating systems and applications. And we launched Splunk for VMware. It’s an application build on top of Splunk that gathers data from from different levels of the VMware virtual stack including the hypervisor configuration, metrics and events, the host operating system, underlying network and guest OS and applications. The application also gives you predefined searches, alerts and reports to troubleshoot and secure your VMware environment. It’s free and you can download it here.


VDC-OS represents a big leap forward in managing the complexity virtualization hoists upon us. Finally vendors like VMware and Microsoft (will soon ship their own System Center Virtual Machine Manager) admit managing complex combinations of virtual resources is difficult and important. This is great for monitoring the hypervisor and virtual guest sessions, but what about the resident guest operating systems or applications? Its still impossible to correlate activity and performance at an application level with resource utilization and performance down to the bare metal

While these vendors are focused on deploying and tracking the resources themselves, Splunk focuses on providing visibility into the complex interactions and dependencies within a virtual infrastructure. Splunk finds, collects and persists the otherwise perishable log, event and configuration data from dynamic virtual instances as they come and go. Splunk correlates data across tiers in the virtual stack — both inside and outside the hypervisor and guests including the physical servers, hypervisor, VMs, and deployed applications,.

When you point your web browser to the Splunk for VMware application you’ll notice several dashboards already created.

• VM Metrics Dashboard - a view of the last hour’s memory and CPU utilization across all running VMs so you can pinpoint hot spots.
• VM Status Dashboard - current configuration, available storage and other key status indicators from different tiers including hypervisor; access & weblogic logs from deployed applications within the guest OS; perfmon, ps and top from the guest OS’s.
• VM Searches Dashboard - all searches, alerts and reports included with Splunk for VMWare.

You’ll see on the searches dashboard a number of investigation searches that correlate the VMWare API data with OS data from within the guests to perform complex investigations in a single step. This dashboard also shows you the details of predefined alerts like looking for guests with heartbeats, looking for storage capacity problems, and other common issues.

As concepts like VMware’s VDC-OS become reality (some time in 2009 according to VMware) having the ability to trace transactions through a virtual infrastructure will become even more important. Every layer of management and abstraction (and yes that’s what virtualization is) means more complexity to manage. Just as with previous VMware products, VDC-OS will not manage physical hardware that has not been virtualized. And understanding how the virtual infrastructure is interacting with non-virtualized servers, storage and networks will remain a critical requirement.

Check out Splunk for VMware and let us know what you think and how we can continue to build on it together.

]]> http://blogs.splunk.com/thebaum/2008/09/19/splunking-vmware-virtualization-at-vmworld/feed/ http://blogs.splunk.com/thebaum/2008/09/12/splunk-in-the-fast-lane-welcome-godfrey/ http://blogs.splunk.com/thebaum/2008/09/12/splunk-in-the-fast-lane-welcome-godfrey/#comments Sat, 13 Sep 2008 04:04:37 +0000 thebaum http://blogs.splunk.com/thebaum/?p=164 Things are moving pretty fast at Splunk and I wanted to comment on the exciting news we announced last week.

In 2004, myself, Erik Swan and Rob Das started Splunk with a vision to battle IT complexity by embracing it. We were thinking of things a bit differently. A different way to address the management of IT by applying search to millions of data center artifacts. Traditionally these artifacts were summarized, filtered and reduced and then forgotten - leaving us humans in a pickle when we needed to figure out what’s really going on. For us Splunk was also about a different way to interact with the market taking an approach of utter transparency. Our public product road maps, freely downloadable software and straightforward marketing had even our early stage venture capital investors thinking we were crazy.

By start-up standards, we seem to have succeeded. Splunk now has more than 250,000 user downloads, more than 750 enterprises, service providers and government agencies worldwide as paying customers and a growing list of partners who embed Splunk into their software, hardware and managed services including companies like Cisco and British Telecom. According to my venture capital friends, very few start-ups make it to where we are today. But, fueled by a love for innovation and so many passionate users we’ve challenged ourselves to see beyond achieving success as a start-up. We believe Splunk can be a company that gets the IT industry thinking differently.

Creating change isn’t easy and we’ll need all the help we can get. Fortunately, we’ve been blessed with an ability to attract top talent at all levels. But our most recent success tops them all. Godfrey Sullivan has joined us as our new President and CEO. When you meet him you’ll realize the incredible passion he has for building great companies. Most recently he was President and CEO of Hyperion Solutions. He took Hyperion over a period of six years to $1B in revenues. Hyperion was acquired by Oracle in 2007 for $3.3B. Godfrey also serves on the board of directors of Citrix Systems, Inc., and Informatica Corporation. Just as important as his business and leadership abilities, Godfrey has the cultural DNA that fits right in at Splunk.

Here’s the yin and yang that is Godfrey. He owns one of only 4,038 1994-1997 Ford GTs. Now this thing is fast, really fast. 0–60 mph (0–96 km/h): 3.3 seconds 0–100 mph (0–160 km/h): 7.3 seconds Standing 1/4 mile: 11.2 seconds @ 134.2 mph Top speed: 212 [11] And his other car is a Toyota Prius. Enough said. Godfrey couldn’t join us at a better time. We’re scaling all aspects of the business and need the leadership of someone who’s been through this type of explosive growth before. For me personally, it’s pretty cool to work beside someone of his experience, talent and steady as she goes outlook on life. And I get to continue to do what I do - build things. I’m now leading the team building our partner ecosystem working with Developers, MSPs, Resellers, Technology Partners and System Integrators around the world.

Of course this hyper growth wouldn’t be possible without your passion and support. Thank you all for that.

Happy Splunking!

]]> http://blogs.splunk.com/thebaum/2008/09/12/splunk-in-the-fast-lane-welcome-godfrey/feed/ http://blogs.splunk.com/thebaum/2008/09/03/situational-awareness/ http://blogs.splunk.com/thebaum/2008/09/03/situational-awareness/#comments Wed, 03 Sep 2008 08:01:23 +0000 thebaum http://blogs.splunk.com/thebaum/?p=154 We’ve been hearing a lot lately about the death of SIEM technologies. But isn’t the question less about a legacy technology dying and more about the dimensions on which the next mass adopted security capability will be born? Clayton Christensen first described a model for disruptive technology in his book The Innovator’s Dilemma and his follow on The Innovator’s Solution. Christensen describes a theory about how disruptive technologies over take sustaining technologies by delivering value on new dimensions that established vendors overlook as unimportant, low end or just don’t think about because they’re too busy improving their legacy. Christensen’s work offers an interest framework to think about what’s taking place in the market for SIEM security management solutions.

Any enterprise trying to secure their IT infrastructures knows the state of the art in SIEM security approaches falls short. And trends like virtualization are making things even more difficult. System and security administrators and analysts are inundated with too many potential incidents and its too difficult and time consuming to investigate even a fraction of them. Achieving a greater comprehension of the meaning of potential incidents and the projection of their status in the near future is the real goal. The idea, called “situational awareness” is often, however, impossible to achieve. We are so dependent on pre-programed rules in our SIEM solutions that we lack the ability to perform our own analysis because the original raw data has been filtered out, thrown away or we have no practical way to make sense of it.

Observation: If the technology is sufficiently complex as to allow the vulnerability to exist, can we really build complex technology to catch all the possible issues or scenarios?

As a reference point see David Hazekamp, Security Architect at Motorola, talk about the importance of retaining all security data across the Motorola global SOC infrastructure and integrating access to all this data into existing SIEM solutions.

Of course reaching this understanding requires one suspends their disbelief about the effectiveness of current SIEM security technologies. Usually this means you’re not a vendor or you’re a vendor with little or no vested interest in current approaches. So with this let’s examine the typical enterprise deployment of security technologies.

Defense in Depth

This is where every good enterprise security architecture starts. In order to begin securing your environment you’ve got to have data, raw data. In most data centers this takes the form of syslog from network devices and servers, SNMP traps, OPSEC or LEA interfaces for firewall events, WMI for Windows desktop and server events, IDS and IPS signature scans and application level firewall examination of common services like FTP, HTTP, SFTP, SCP etc. The thinking is you need to look at everything. Perhaps you’ll even want to pull in information from physical security systems like badge readers.

Security Information Management (SIM)

The next step in the process is to manage all this raw data and filter it down to a manageable number of events, traps and alerts. Collecting, storing and providing some basic analysis on all this data is the job of a SIM. Typically, as Raffy points out, the data is parsed, normalized and stored in a structured RDBMS. Parsing, normalizing and structuring all this data is great if the data doesn’t change or you don’t have too much of it. But if you’re dealing with data formats that aren’t static or you’re trying to store terabytes of this data an RDBMS won’t be your friend.

Security Event Management (SEM)

Once a SIM has done it’s job you’re ready to aggregate, correlate and start reporting on potential incidents using a SEM to do the job. SEM’s usually consist of lots of rules that look for combination and patterns of events indicating that a possible attack or breach may be underway. Essentially the SEM rules attempt to codify what we humans know about vulnerabilities in our IT systems and possible ways to exploit them. The goal is to provide some real-time information usually in the form of reports, dashboards and visualizations to operations and security analysts who work to keep the infrastructure secure.

Situational Awareness (SA)

SIEM correlation can be interesting for discovering a pattern or related event but the ability to work an issue outside of these “canned” rules and events becomes the real problem. Unfortunately, what all to often happens is there are so many possible attacks, operations and security staff are overwhelmed with potential incidents to investigate and not every event or pattern of interest is going to be discovered via the pre-built rules. Situational awareness is the attempt to perceive environmental elements within a volume of space and time. Comprehension cannot be achieved if the data being bubbled up is filtered according to a set of rules and the technology does not allow a human to perform their own analysis of the raw data as generated by the environment itself. All technologies have their weaknesses and those that perform correlation are no different.

Thus whilst canned SIEM correlation provides value in bubbling things up — we still need the ability to dig into the raw data to fully perceive and comprehend what is taking place. Now mind us all SA is not a new concept. It has been applied rather robustly by decision-makers in complex, dynamic areas from aviation, air traffic control, power plant operations, military command and control — to more ordinary but nevertheless complex tasks such as driving an automobile or motorcycle. And yes it has been mentioned before in security operations, particularly in government agencies.

Situational awareness is a simple as, “I discovered a problem and need context.” Whether discovery comes from a operational log, a security event log, a SIEM correlated events or aggregated events, a telephone call or something read on a blog. The ability to access and quickly analyze the raw data from the far reaches of your IT environment is the only true path to situational awareness. The idea extends well beyond log and event management and is an enabler for Operations and Security best practices alike where questions are answered by attaining context around an event. It should not be limited by the structure of the data or the structure of the queries and reports that the vendor provided.

I’m not sure if Raffy is right and SIEM is dead yet, but for certain it will eventually become just one part of a more comprehensive, flexible and human enabled ways of securing our IT infrastructures.

]]> http://blogs.splunk.com/thebaum/2008/09/03/situational-awareness/feed/ http://blogs.splunk.com/thebaum/2008/08/25/man-versus-machine-part-one/ http://blogs.splunk.com/thebaum/2008/08/25/man-versus-machine-part-one/#comments Mon, 25 Aug 2008 09:05:33 +0000 thebaum http://blogs.splunk.com/thebaum/?p=125

Recently I gave a talk at the BT annual technology gathering. The setting was a really beautiful estate called The Grove just north of London in Hertfordshire England. A couple hundred of BT’s smartest technology managers were in attendance and I was supposed to think of something to hold their interest for an hour. I got to thinking about all the technology and infrastructure BT must have and how in the world do they manage it. I started gathering data. With internal growth, new projects like BT’s 21st Century Network and acquisitions over the past decade through BT Global Services outsourcing contracts the company has a lot of IT infrastructure.

• 74 data centers,
• 163 countries,
• 3,000 applications,
• 6,000 different types of systems/devices and
• 17,000 IT staff (6,000 BT and 11,000 outsourced).



I also spent a few hours with some of BT’s brightest architects who are working on attempts to virtualize every layer of their infrastructure — network, storage, database, application, web servers, VoIP, collaboration, ordering, billing, provisioning, monitoring etc. What’s their biggest problem I asked. Resoundingly it was “our customers are still often the ones that tell us stuff is broken.” This was so reminiscent of my time at places like Yahoo! where we’d have these 7×24 war rooms during key outages and the daily conference calls with 30-40 people on the line all emailing logs and configurations to each other.

As our IT infrastructures become incredibly complex, dynamic, service oriented, virtualized and mission critical we’re confronted with this battle raging in our data centers. And it appears the machines are winning and the humans are losing.

Our biggest problem is figuring out — did something go wrong? Why? Where does truth lie? According to market researcher IDC In 2007 > $140B spent managing the world’s data centers. IT OPEX is growing at 2.5 times the rate of hardware spend and 1/3-1/2 of TCO is spent recovering from problems. The cost of availability now dwarfs the purchase and maintenance cost of technology.

So what have we as an IT industry done to address the problem?

We’ve created concepts like ITIL and CMDBs. While there are some good processes improvements here for sure, these top down modeling approaches and pre-determined rules only tell us what we already know. In my experience it is not the things we already know about that bite us in the ass and take our systems down for prolonged periods of time. It’s the multitude of unanticipated and unavoidable dependencies and interactions that take place in an complex system. And it’s impossible to know what set of dependencies and interactions will cause downtime until it occurs. Our infrastructures are just too indeterminate. That’s the point after all. Tier it, load balance it, virtualize it. So we don’t have to worry about the dependencies and interactions among all the different components. Well guess what? We do have to care. Because we have to fix it when it goes wrong.

Take the analogy of a complex air traffic control system. Sure the air traffic controllers feel really great when they arrive at work in the morning. They’ve got their coffee, flight plans and a good handle on the early morning inbound and outbound traffic.



Then the day gets a bit more challenging. Weather conditions over Chicago backs up landings at O’Hare. A baggage handler and mechanic strike slows down JFK departures. A pilot radios he’s three degrees north over Pennsylvania but where is he really? Now you need radar. Throw the flight plans out the window. You needs to know what’s actually happening now.



So how do we establish the equivalent of radar for a complex IT infrastructure. Component monitoring doesn’t work any more. If the problem is a single component failure, we already know about it. We’ve already automated the swapping in of a new machine or device. And we can reboot software components automatically. IBM’s has their own marketing play on this called “Autonomic Computing” but that too seems to only focus on the simple single component issues not the indeterminate chaos that ensues in a real running system. And it seems like more slideware than real solutions.

In my next post I’ll tackle the issue of how we might look at things differently.

Stay tuned.

]]> http://blogs.splunk.com/thebaum/2008/08/25/man-versus-machine-part-one/feed/ http://blogs.splunk.com/thebaum/2008/08/15/splunk-live-southwest-2008/ http://blogs.splunk.com/thebaum/2008/08/15/splunk-live-southwest-2008/#comments Fri, 15 Aug 2008 18:06:41 +0000 thebaum http://blogs.splunk.com/thebaum/?p=126

This week we’ve been moseying through the Southwestern part of the US with our Splunk Live show. We changed up the format a bit with Splunk technical workshops in the morning and customer round tables in the afternoon. The technical workshops were a big hit with more than 200 people registered to engage with our Splunk Experts. During the workshop you were able to download, install, configure and start using Splunk on your laptop or server with remote access. The best part about Splunk Live events though is sharing ideas with other Splunk fanatics.

Ryan Peterson from Infusionsoft, a marketing automation company, gave a great talk in Scottsdale about his Splunk deployment for the company’s email infrastructure. Ryan is tasked with keeping more than 12M emails a week flowing out of the system to support Infusionsoft’s Automated Follow-up Technology (AFT). Ryan has multiple servers in different geographies in addition to PCI Compliance requirements. He demonstrated using Splunk to troubleshoot problems spread across the messaging infrastructure, address reporting inaccuracies and deliver PCI reports to auditors. He’s even indexing the content of email with Splunk using a scripted LDAP data input. Cool stuff.

In San Diego Tony Doan of the Genomics Institute at the Novartis Research Foundation (GNF) and Eric Van Johnson from Sony Consumer Electronics joined us. Tony is a security engineer and former pen tester. He also confesses to be a recovering Unix sysadmin. GNF has 600 Windows desktops and several hundred Windows and Linux servers supporting the discovery of new biological processes and improved human therapeutics. Tony discussed how they splunk Cisco CSC, Bluecoat, Symantec AV, Arpwatch, Cisco Switches and Wifi access points to find what he calls “previously unknowns” to improve operational availability and security. He says they’re finding new uses everyday but Tony’s favorite is splunking Cisco IPS and Cisco MARS events looking for odd behaviors. Next up for GNF is eating Windows Event Logs and Windows Registry inputs together with summary indexing for consolidated reporting.

Eric Van Johnson is the eServices Hosting and Operations Manager at Sony Consumer electronics. He led an great discussion on splunking IBM Websphere and MQ Series events including how Sony has integrated operations and development environments to identify problems with complex apps more quickly and avoid unnecessary escalations to the development team. He shared with us Sony’s roll out of Splunk to their Business Intelligence Group. The idea is to complement aggregated WebMethods data reporting for business activity monitoring. Next up he wants to feed Splunk data back and forth with Verizon’s hosting operations since some of the Sony servers are hosted at Verizon and Verizon is also using Splunk.

In LA Rich Horace, Director of Systems Engineering and Operations at Fox Interactive Media demonstrated how Fox uses Splunk in the Fox Audience Network. Basically these are the guys that serve web advertisements across all the Fox properties including MySpace, Rotten Tomatoes, Fox Sports and IGN. He’s challenged with launching new monetization platforms and keeping the existing ones running. Rich gave a fantastic overview of his Splunk installation which consolidates/aggregates data form disparate systems in order to protect against hackers and meet PCI and SOX requirements. He currently runs an environment with ~600 Linux servers, load balancers, servers, NetApps and network switches. So far he’s indexed 1.5B events. We engaged with everyone in a lively discussion about securing production sites from developers and controlling and auditing access to data using Splunk’s access controls and search filters. Rich also discussed how Fox is using Splunk to integrate with various Citrix products including Netscaler and XenApp.

Thanks to everyone who shared their stories with us this week, it was really awesome.

]]> http://blogs.splunk.com/thebaum/2008/08/15/splunk-live-southwest-2008/feed/