I feel like I should post a follow-up to my recent post about SIM is dead. Here are some points I would like to clarify:

• If I talk about SIM or SIEM, I am talking about the way current SIM solutions are working and the way they are implemented. That means things like relational database, fixed schema, parsed and normalized data, or hierarchical scaling.
• Do I really believe that SIM is not useful? No. And I am not just saying that because I own stock in a SIM company. Just like Alex says in a comment on my original blog entry: IDS is not dead. SIM is probably not dead either. I know of quite some people that are very happy with their SIM implementation. However, there are many limitations with the way today’s SIMs are architected.
• The architectural limits cripple the SIMs. They cannot deal with really large event volumes. With the current threat landscape this means that many use-cases cannot be implemented with a SIM. They simply can’t scale to that extent. Leverage IT search to do the heavy data lifting.
• Network world published a review of recent SIEM technology. They note correctly that application data is becoming more and more important. SIMs have traditionally been built for firewalls, intrusion detection systems, and vulnerability scans and that’s what they are really good at. To be precise. That’s where some SIMs are really good. But as soon as you are dealing with other data sources, such as call detail records (CDRs) or other crazy application logs, you start overloading the existing schema, apply one hack after the other and eventually cripple the entire system.
• Some SIMs have done a great job of implementing features that are well-suited for security operations centers (SOCs). In these environments, analysts are working on a console 7×24. They need features like workflow, collaboration, ticketing, live channels, etc. In such an environment, a collaborative approach between a SIM and an IT search solution can be quite effective. IT search is dedicated to data management, data routing and collection, and forensic investigations, as well as reporting. The SIM can be dedicated to real-time correlation, collaboration, and providing a front-end for the analysts.

This should clarify some of my points.

What do you do if your system administrator locks you out of your critical systems, changes the root password and then quits? If you haven’t thought about this, you are not the only one. San Francisco officials are facing exactly that question. A disgruntled employee locked out all the system administrators from some fairly critical systems, as you can read in the San Francisco Chronicle.

Insider crime is an area in computer security that still doesn’t get much attention. One of the problems is that the frequency of incidents is fairly low and therefore the problem rates low on a company’s charter. However, the big problem is that the average cost of such an incident is really high. In reality, companies are still struggling with protecting their perimeter. They are worried about outside attackers, script kiddies, about their competition breaking in, attacks of Chinese hackers, Russian crime rings, etc. They should balance their efforts to protect from these threats as well as from malicious insiders.

In this specific case, there were some very obvious signs that should have been noticed. The employee should have been on a watch-list and his activity should have been under review. He was about to be fired. This should have put him into a group of people that are monitored closely. Monitoring is not easy. It is all about people and processes and a little bit about technology. There is unfortunately no software or security tool out there that could detect an insider. And there will never be one.

As I point out in my book, you need to define a process that classifies employees. People on a watch list need to be monitored more closely.  Audit records need to be recorded, especially for privileged activities (such as the ones executed by system administrators). Those records then need to be stored in a place where nobody can tamper with them (for example in Splunk). The records then need to be reviewed on a regular basis. Hopefully by a separate team. Ideally the reviews are automated to ease the work load (for example through alerts in Splunk).

A second step has to be the implementation of proper security processes. Separation of duties, for example. The system administrator by himself should not be able to alter all the passwords necessary to access a system. In reality, this is really hard to enforce. However, if the preventative control cannot be enforced, a detective control should be put in place. Firstly, system logs should be centrally collected and analyzed, and secondly, the file systems should be monitored for changes. That way, all changes can be reviewed to see what the exact impact of Terry’s actions was.

Traditional computer security attacks are violating policy. Specialized sensors can be developed and deployed to monitor for signs of attacks. Insider crime is often executed without violating any policy. For example, a system administrator has the right to change passwords. However, as in San Francisco’s case, Terry abused that privilege to lock everybody out of the machines. The net is that one has to monitor not just violations or obvious attacks, but also regular and seemingly benign activity. This results in a huge amount of data from a lot of different sources. Make sure you have a solution that can deal with all of it.

An Interesting side fact: The department of technology is worried about a third-party accessing the systems with Terry’s account. This is definitely the time where Splunk needs to be in place to monitor all the records to check for any account access. This information can then be used by law enforcement to take action.

This article: “San Francisco Hack: Where Was the Oversight?” contains some of my comments about the case.