I feel like I should post a follow-up to my recent post about SIM is dead. Here are some points I would like to clarify:

• If I talk about SIM or SIEM, I am talking about the way current SIM solutions are working and the way they are implemented. That means things like relational database, fixed schema, parsed and normalized data, or hierarchical scaling.
• Do I really believe that SIM is not useful? No. And I am not just saying that because I own stock in a SIM company. Just like Alex says in a comment on my original blog entry: IDS is not dead. SIM is probably not dead either. I know of quite some people that are very happy with their SIM implementation. However, there are many limitations with the way today’s SIMs are architected.
• The architectural limits cripple the SIMs. They cannot deal with really large event volumes. With the current threat landscape this means that many use-cases cannot be implemented with a SIM. They simply can’t scale to that extent. Leverage IT search to do the heavy data lifting.
• Network world published a review of recent SIEM technology. They note correctly that application data is becoming more and more important. SIMs have traditionally been built for firewalls, intrusion detection systems, and vulnerability scans and that’s what they are really good at. To be precise. That’s where some SIMs are really good. But as soon as you are dealing with other data sources, such as call detail records (CDRs) or other crazy application logs, you start overloading the existing schema, apply one hack after the other and eventually cripple the entire system.
• Some SIMs have done a great job of implementing features that are well-suited for security operations centers (SOCs). In these environments, analysts are working on a console 7×24. They need features like workflow, collaboration, ticketing, live channels, etc. In such an environment, a collaborative approach between a SIM and an IT search solution can be quite effective. IT search is dedicated to data management, data routing and collection, and forensic investigations, as well as reporting. The SIM can be dedicated to real-time correlation, collaboration, and providing a front-end for the analysts.

This should clarify some of my points.

What do you do if your system administrator locks you out of your critical systems, changes the root password and then quits? If you haven’t thought about this, you are not the only one. San Francisco officials are facing exactly that question. A disgruntled employee locked out all the system administrators from some fairly critical systems, as you can read in the San Francisco Chronicle.

Insider crime is an area in computer security that still doesn’t get much attention. One of the problems is that the frequency of incidents is fairly low and therefore the problem rates low on a company’s charter. However, the big problem is that the average cost of such an incident is really high. In reality, companies are still struggling with protecting their perimeter. They are worried about outside attackers, script kiddies, about their competition breaking in, attacks of Chinese hackers, Russian crime rings, etc. They should balance their efforts to protect from these threats as well as from malicious insiders.

In this specific case, there were some very obvious signs that should have been noticed. The employee should have been on a watch-list and his activity should have been under review. He was about to be fired. This should have put him into a group of people that are monitored closely. Monitoring is not easy. It is all about people and processes and a little bit about technology. There is unfortunately no software or security tool out there that could detect an insider. And there will never be one.

As I point out in my book, you need to define a process that classifies employees. People on a watch list need to be monitored more closely.  Audit records need to be recorded, especially for privileged activities (such as the ones executed by system administrators). Those records then need to be stored in a place where nobody can tamper with them (for example in Splunk). The records then need to be reviewed on a regular basis. Hopefully by a separate team. Ideally the reviews are automated to ease the work load (for example through alerts in Splunk).

A second step has to be the implementation of proper security processes. Separation of duties, for example. The system administrator by himself should not be able to alter all the passwords necessary to access a system. In reality, this is really hard to enforce. However, if the preventative control cannot be enforced, a detective control should be put in place. Firstly, system logs should be centrally collected and analyzed, and secondly, the file systems should be monitored for changes. That way, all changes can be reviewed to see what the exact impact of Terry’s actions was.

Traditional computer security attacks are violating policy. Specialized sensors can be developed and deployed to monitor for signs of attacks. Insider crime is often executed without violating any policy. For example, a system administrator has the right to change passwords. However, as in San Francisco’s case, Terry abused that privilege to lock everybody out of the machines. The net is that one has to monitor not just violations or obvious attacks, but also regular and seemingly benign activity. This results in a huge amount of data from a lot of different sources. Make sure you have a solution that can deal with all of it.

An Interesting side fact: The department of technology is worried about a third-party accessing the systems with Terry’s account. This is definitely the time where Splunk needs to be in place to monitor all the records to check for any account access. This information can then be used by law enforcement to take action.

This article: “San Francisco Hack: Where Was the Oversight?” contains some of my comments about the case.

Pretty much exactly 5 years ago, in June 2003, Gartner declared Intrusion Detection Systems to be dead. Before Gartner can do so, I will state that SIM is dead.

The crime landscape has shifted. We used to be worried about network layer attacks, TCP/IP attacks where funky flags were crashing your systems. This is gone. We really don’t worry about them anymore. We have systems to stop these attacks. The crime has shifted up to the application layer. There are attacks over instant messaging, there are SQL injections, there are application layer attacks. You have to start monitoring the application layer. Compliance requirements are shifting too. For example, the PCI DSS 1.1 requires the usage of application layer firewalls by June 2008. Applications need to be verified for vulnerabilities and not just the platform.

Some of the problems I see with Security Information Management are (the first four are adapted from the Gartner IDS press release):

• False positives in correlation rules
• Burden on the IS organization by requiring full-time monitoring
• A taxing incident-response process
• An inability to monitor events at rates greater than 10.000 events per second
• High cost of maintaining and build new adapters
• Complexity of modeling environment

However, the biggest problem lies in the fixed event schema. SIMs were built for network-based attacks. They are good at dealing with firewall, IDS, and maybe vulnerability data. Their database schema is built for that. So are the correlation rules. Moving outside of that realm into application layer data and other types of logs can get hard. Fields don’t match up anymore and the pre-built correlation rules don’t fit either.

We need a new approach. We need an approach that can deal with all kinds of data. An approach that deals with multi-line messages, with any type of fields, even with entire files as entities. There is a need for a system that can collect data at rates of 100.000 events a second and still perform data analysis. It needs to support large quantities of analytical rules, not just a limited set. The system needs to be easy to use and absorb knowledge from the users.

The solution is called IT search.

Yesterday marked yet another milestone in my life as an author. I got the first 5 chapters of my book back from production. The Applied Security Visualization book is slowly coming together. After working on the book for one and a half years, it is great to finally see how the book is going to look. The graphs are placed on the pages and the layout is done. It finally feels like a real book. The book will be out by BlackHat at the beginning of August.

.

.

You can pre-order the book on Amazon. It is about 400 pages and contains the following chapters:

1. 1. Visualization

2. 2. Data Sources

3. 3. Visually Representing Data

4. 4. From Data to Graphs

5. 5. Visual Security Analysis

6. 6. Perimeter Threat

7. 7. Compliance

8. 8. Insider Threat

9. 9. Data Visualization Tools

The book ships with a live visualization CD. DAVIX, the data analysis and visualization UNIX, contains all the visualization tools discussed in chapter 9. They are all readily installed so you can use them to visualize your own data. No need to go through any crazy installation processes. The Web site for DAVIX is going to be ready by BlackHat, where we will officially launch DAVIX. If you are interested in a pre-version, drop me an email.

The payment card industry data security standard, PCI DSS for short, was developed by the credit card industry to address data theft. The standard consists of twelve security requirement. Anything from traffic policies to requirements around anti virus software are covered by the standard.

If you are a company that does more than 20.000 transactions per year, you will have to implement the twelve requirements. If you are doing less, you will get away with a quarterly vulnerability scan.

IT search, Splunk, can directly address some of the areas and indirectly address most of the others. Specifically the areas where IT search assists are the following:

• Log management (PCI requirement 10)
• Secure & Central Log Collection (PCI requirement 10.5)
• Audit Trail Retention (PCI requirement 10.7)
• Daily Log Review (PCI requirement 10.6)
• Secure Remote Access (PCI requirement 7.1)
• File Integrity Monitoring (PCI requirements 10.2.2, 11.5 and 10.5.5)
• PCI Control Reporting*

The Splunk for PCI application can be downloaded from SplunkBase. It provides a set of 91 searches and 57 reports, a dashboard, and a set of alerts that can be used to monitor the control objectives. The application makes use of Splunk’s IT search capabilities to address PCI. IT search has some very unique capabilities and is uniquely positioned to address PCI compliance:

• satisfy ad-hoc requests form auditors
• do large-scale reporting and investigations
• automate control objective monitoring
• add new control objectives and policies that require flexible monitoring and correlation capabilities
• support ever changing data sources
• re-use already collected data
• incorportate file monitoring (not just traditional one-line log messages)

The Splunk for PCI application also gives you a capability to implement compensating controls for some of the PCI requirements. Also make sure to check out the daily log review process that helps you very easily tackle requirement 10.6.

Splunk is serious about PCI compliance: We are now part of the PCI Council. This is going to ensure that we know about upcoming changes to the PCI standard ahead of time and we can help influence future direction of it.

This morning, there was yet another case of phishing that was reported by the New York Times. This phishing incident, Larger Prey Are Targets of Phishing, is interesting because of the victim demographics: executives of large companies. As I just learned, this is also referred to as whaling. We have all seen phishing emails that tried to lure us into logging into our PayPal account. But an email from the United States District Court in San Diego that has a very authentic look is a different story. Would you fall for it?

The best way to address phishing is to educate users to make sure they don’t give out personal information. Have a look at the AntiPhishing Working Group’s phishing checklist that contains a lot of specific tips to prevent successful phishing attacks.

Splunk can addresses a couple of use-cases surrounding phishing attacks:

• Detecting, after the fact, whether someone in your company fell victim to the scam (phishing).
• Protecting your company from being phished. (In today’s story, the United States District Court in San Diego)

Detecting Phishing Victims

Once you know about a phishing attack, you can use Splunk to figure out whether anyone in your company has fallen victim. There are a few ways to do so, depending on the attack vector:

1. The phish infects the victim and installs a trojan that starts leaking information.
2. The phish uses a Web site to collect victims’ personal information (such as credit cards)

Both of these infections will start communicating with the outside. In the case of the phish reported today, the computers started communicating with machines in Singapore. By analyzing the traffic patterns and figuring out where in the world connections are being made to, this infection can be detected very easily. The Splunk reporting is a great way to quickly generate traffic reports and isolate traffic patterns based on geographic locations of the communicating machines. If , for example, your normal access pattern looks like the first graph and then after some time, you get the result of the second picture, where China suddenly shows up at second position, there might be something wrong.

Normal traffic patterns hitting Web site:

Suspicious traffic pattern hitting Web site. Note China on second position:

Protecting Your Company From Being Phished

If you are operating a Web site, you should try to make sure that there is nobody trying to phish it. There are a couple of ways that IT Search can help you with this:

• Monitor your Web server logs for non-complete session requests. A lot of phishers request images from your site, but not the original site itself (the HTML page).
• Monitor Web server logs for sessions that directly send a login, without ever requesting the login page itself. This happens when the victim logged into the phishing site and the credentials are passed to the real site, making everything look normal for the victim.
• Check DNS lookups and see whether you get a lot of lookups from one single machine. This is tricky and you need to know the baseline of lookups, but spikes might turn out interesting to investigate.

Here is a search in Splunk that you can use to determine whether someone posted credentials without ever requesting the login page:

sourcetype=access_comined (login_form.php OR sales.php) | stats count by clientip | search count=1

This assumes you have a page, sales.php, which you can only access once you logged in via the login_form.php. For more complicated Web site architectures, you will have to build a more sophisticated search that uses transactions, but more on that another time.

I was giving a talk at SOURCEBoston 2008. The topic this time was around general visualization and what has gone wrong in security visualization in the past. I showed how we can learn and steal from other disciplines, in this case, the New York Times. The NYT has done some pretty fantastic work in the area of data visualization. Their interactive market map, for example, is a great way of exploring stock data. During the talk, I outlined some of the design principles that the NYT graphics department is using when they are designing their graphs: Show - Don’t Tell.

To start my presentation, I showed a little video about security visualization (see below).

At conferences lately, I find myself not to be the only one that talks about security visualization. More and more presentations are showing visualizations. A lot of projects are using visualization to help them analyze all the data at hand. At SOURCE, Dave Dittrich from the University of Washington, talked about BotNet analysis and visualizing network traffic captured from BotNets. He definitely has a challenge of displaying large amounts of data. We discussed some approaches and possibly, parallel coordinates, could work for his data. Parallel coordinates are what I used in my book for some BotNet traffic analysis.

As part of the common event expression (CEE) effort, a list of field names has been published.

If log records from different log sources have to be correlated or reports have to be generated across different log sources, a common set of field names is needed. Take a firewall log example. Assume that you have two types of firewalls in your environment: Netscreen and PIX. Both devices write different types of log entries. Assume you have a parser that extracts fields from the two logs. Each of the parsers might call fields differently, making it either impossible, or really hard to correlate these two log files. Just think about reporting. How do you find the top source addresses across both logs? These are logs from each of the firewalls:

Netscreeen:

May  5 17:01:40 45.2.0.1 NOC-FWa: NetScreen device_id=NOC-FWa [Root]
system-notification-00257(traffic): start_time=”2006-05-05 17:01:40″
duration=0 policy_id=52 service=tcp/port:26212 proto=6 src zone=backbone
dst zone=noc-mgt action=Deny sent=0 rcvd=0 src=222.81.119.59dst=45.2.121.102
src_port=7000 dst_port=26212

Pix:

Jan 18 12:43:50 192.168.1.1 %PIX-6-106015: Deny TCP (no connection)
from 208.58.193.69/1062 to a.b.c.d/443 flags ACK

If you report on “src”, you won’t get the “from” from the PIX log. We need unified names.

It is not just important to have a common set of names, but also a common understanding of what individual fields mean. What is the semantics of a field? For example, how do you measure a duration? In seconds? Hours? Days? What is a destination host? Is it fully qualified or just the host name itself? The field list, which can be found in this post: CEE Fields List is a first step towards standardizing this.

Note that, for example, ArcSight’s CEF publishes a dictionary along with their log syntax. The CEE field list can be used to standardize the names across various log formats and can hopefully substitute and expand ArcSight’s dictionary.

The common event expression (CEE) effort is moving along. If you haven’t seen much coming out of CEE, it is not that we are not working on it. We have been busy defining and hashing out various aspects of the CEE standard. I am getting ready to release a list of fields for the syntax part of CEE. The taxonomy is moving along as well and I am compiling the final pieces to release for discussion.

If you are interested in the public discussions around CEE, the Mailing list archives are now online.

For the past year I have been working on a book about visualization. It will be called “Applied Security Visualization“. The book is going to talk about all the aspects of visualizing security data. Anything from important data sources and graphs to use-cases and open source tools for visualization. The main use-cases I write about evolve around Perimeter Threat, Compliance, and Insider Threat.

Last year during RSA, Addison-Wesley (my publisher) recorded some videos, where I talk about the book and some of its contents. Here are the links to the videocasts:

• Security visualization
• Bridging security and visualization

At this point, I have one more chapter to write before the book is done. A rough-cut version should be available by RSA this year and the book should be out by BlackHat (August). Keep your fingers crossed!