At RSA this week it’s easy to got lost in the menagerie of security technologies to conquer malware proliferation, stomp out spam and protect virtualized and cloud computing environments. But the most recent statistics show we are still losing the war on cybercrime. Symantec’s latest Internet Security Threat Report sited 1,656,227 malicious-code threats last year and 75,158 new active bot-infected computers per day. And yes the United States is still the most frequently targeted by denial-of-service attacks accounting for 51% worldwide and the top country for underground economy servers advertising stolen credit cards accounting for 67% of all activity worldwide.

Why are we losing so badly? Not surprisingly, there was a lot of talk at RSA about the Conficker worm. Some of the chatter points to reasons why the security industry is falling behind. At first glance, the Conficker worm looks harmless. So far there are not too many significant reports of infected machines and hijacked data,
but it may be too early to feel so smug about it. The worm’s real danger is its demonstrated ability to evade the expensive IDS technology enterprises have put into place and rely on today. Estimates are that 90% of the enterprise IDS implementations have failed to detect the worm’s presence and create some kind of actionable alert. How can this be?

Conficker properties are simple but different from the typical threat. First Conficker affected systems outside of IDS coverage like USB keys and mobile user laptops. So if you’re looking for attacks from outside your network only, you won’t see it. It’s a “walk-in virus”. Second it isn’t greedy like Code Red and other viruses of late. The Conficker worm has built-in sleep cycles. So where a typical worm might scan 1,000 or 10,000 IPs a minute, Conficker was happy to scan maybe say 100 and evade the baseline trip wires. Third Conficker is very selective with its payload delivery. It only delivers when it sees a vulnerability. All this helps Conficker evade IDS systems that want to witness the crime. But Conficker is the perfect crime in that it goes undetected. With no payload delivered and seemingly fewer IPs scanned there is no grossly abnormal behavior to witness. The evidence is circumstantial.

At a lunch on Wednesday, Tom Le of BT gave a good overview of how BT Managed Security Services detected Conficker for their customers. It was one of the first times I’ve really been sold on a managed security service beyond the value of cost and convenience.

First, as Tom explained it, they started by assuming IDS would miss the attack. They didn’t assume a payload had to be delivered and didn’t assume that large number of scans were needed to indicate the presence of an intruder. Instead of depending on IDS, BT uses logs and events to baseline the natural behavior of even netbios triggered scans (which Conficker happened to use) and was able to alert on small changes in scans that would be missed if you were only looking at things like netflow. As it turns out most firewalls blocked the netbios scans going out so again most customers didn’t even know they had the Conficker worm present.

Second Tom and his team assumed some type of command and control activity associated with Conficker. They followed the money watching for things like confikur trying to phone home in different ways. By having a broad set of logs and events from switches, routers, applications and IDS they were able to look for outlying behaviors like DNS lookups to obscure locations not typically seen in customer networks and aggregate this information across customers to identify common abnormalities. Tom estimates that BT sees roughly five billion messages a week across their customer base. That’s a lot of messages.

After listening to all the chatter about Conficker and walking the show floor, it gets easier to understand how criminals continue to evade the security infrastructure enterprises put in place. There are just too many ways in which breaches can occur and there is just too much data scattered about to collect and correlate in order to find the anomalies. So the security industry continues down the path of specific solutions to specific vulnerabilities and criminals continue to create new threats that evade the industry’s point approaches. I say the industry as a whole needs to move to more of an adaptable and flexible approach that can apply security to what ever threats arise, when they appear.

The best real world detectives are able to piece together seemingly circumstantial evidence and sift out the clues that lead to catching criminals. But every time it’s different. Perhaps we need to take the same approach in order to obtain more adaptable security solutions. Assume every time it’s different not the same.

Logging broadly and analyzing deeply is one of the best defenses. Without a broad swath of data you won’t have the pieces of the puzzle to put together at the moment you need to solve the crime.

Few criminals are caught in the act.

We’ve been hearing a lot lately about the death of SIEM technologies. But isn’t the question less about a legacy technology dying and more about the dimensions on which the next mass adopted security capability will be born? Clayton Christensen first described a model for disruptive technology in his book The Innovator’s Dilemma and his follow on The Innovator’s Solution. Christensen describes a theory about how disruptive technologies over take sustaining technologies by delivering value on new dimensions that established vendors overlook as unimportant, low end or just don’t think about because they’re too busy improving their legacy. Christensen’s work offers an interest framework to think about what’s taking place in the market for SIEM security management solutions.

Any enterprise trying to secure their IT infrastructures knows the state of the art in SIEM security approaches falls short. And trends like virtualization are making things even more difficult. System and security administrators and analysts are inundated with too many potential incidents and its too difficult and time consuming to investigate even a fraction of them. Achieving a greater comprehension of the meaning of potential incidents and the projection of their status in the near future is the real goal. The idea, called “situational awareness” is often, however, impossible to achieve. We are so dependent on pre-programed rules in our SIEM solutions that we lack the ability to perform our own analysis because the original raw data has been filtered out, thrown away or we have no practical way to make sense of it.

Observation: If the technology is sufficiently complex as to allow the vulnerability to exist, can we really build complex technology to catch all the possible issues or scenarios?

As a reference point see David Hazekamp, Security Architect at Motorola, talk about the importance of retaining all security data across the Motorola global SOC infrastructure and integrating access to all this data into existing SIEM solutions.

Of course reaching this understanding requires one suspends their disbelief about the effectiveness of current SIEM security technologies. Usually this means you’re not a vendor or you’re a vendor with little or no vested interest in current approaches. So with this let’s examine the typical enterprise deployment of security technologies.

Defense in Depth

This is where every good enterprise security architecture starts. In order to begin securing your environment you’ve got to have data, raw data. In most data centers this takes the form of syslog from network devices and servers, SNMP traps, OPSEC or LEA interfaces for firewall events, WMI for Windows desktop and server events, IDS and IPS signature scans and application level firewall examination of common services like FTP, HTTP, SFTP, SCP etc. The thinking is you need to look at everything. Perhaps you’ll even want to pull in information from physical security systems like badge readers.

Security Information Management (SIM)

The next step in the process is to manage all this raw data and filter it down to a manageable number of events, traps and alerts. Collecting, storing and providing some basic analysis on all this data is the job of a SIM. Typically, as Raffy points out, the data is parsed, normalized and stored in a structured RDBMS. Parsing, normalizing and structuring all this data is great if the data doesn’t change or you don’t have too much of it. But if you’re dealing with data formats that aren’t static or you’re trying to store terabytes of this data an RDBMS won’t be your friend.

Security Event Management (SEM)

Once a SIM has done it’s job you’re ready to aggregate, correlate and start reporting on potential incidents using a SEM to do the job. SEM’s usually consist of lots of rules that look for combination and patterns of events indicating that a possible attack or breach may be underway. Essentially the SEM rules attempt to codify what we humans know about vulnerabilities in our IT systems and possible ways to exploit them. The goal is to provide some real-time information usually in the form of reports, dashboards and visualizations to operations and security analysts who work to keep the infrastructure secure.

Situational Awareness (SA)

SIEM correlation can be interesting for discovering a pattern or related event but the ability to work an issue outside of these “canned” rules and events becomes the real problem. Unfortunately, what all to often happens is there are so many possible attacks, operations and security staff are overwhelmed with potential incidents to investigate and not every event or pattern of interest is going to be discovered via the pre-built rules. Situational awareness is the attempt to perceive environmental elements within a volume of space and time. Comprehension cannot be achieved if the data being bubbled up is filtered according to a set of rules and the technology does not allow a human to perform their own analysis of the raw data as generated by the environment itself. All technologies have their weaknesses and those that perform correlation are no different.

Thus whilst canned SIEM correlation provides value in bubbling things up — we still need the ability to dig into the raw data to fully perceive and comprehend what is taking place. Now mind us all SA is not a new concept. It has been applied rather robustly by decision-makers in complex, dynamic areas from aviation, air traffic control, power plant operations, military command and control — to more ordinary but nevertheless complex tasks such as driving an automobile or motorcycle. And yes it has been mentioned before in security operations, particularly in government agencies.

I love “log management.” I hate log management.

I love log management because years ago it was the impetus for IT to move beyond simple SNMP monitoring to collecting and trying to understand a much richer set of data about complex environments.

I hate log management for over the years it has been co-opted by vendors and analysts who’ve pigeon holed it into yet another IT management silo. These vendors and analysts have narrowly defined log management as the collection and storage of logs in some locked repository used to generate static reports to satisfy regulators, auditors and IT governance boards.

Why am I so bitter?

First it turns out logs are critical to many other stakeholders in the enterprise. Operations needs real time access to logs in order to find and fix problems and improve mean time to recovery (MTTR). Security needs logs to catch bad guys. Business people need logs to understand customer and service behavior and provide service level measurements. So locking up logs in a static repository designed for one constituency severely limits their value and diminishes the return on investment not only in a log management solution but also the return on your IT assets overall.

Secondly logs alone don’t provide anyone of the IT stakeholders with a complete picture.

Let’s take a simple example right from the hottest compliance use case today — PCI. The Payment Card Industry (PCI) Security Standards Council founded by American Express, Discover Financial Services, JCB International, Mastercard and Visa has outlined requirements for security management, policies, procedures, network architecture and software design. If you are a merchant accepting credit or debit cards and you process more than 20,000 transactions per year there are twelve specific requirements. Failure to comply with the requirements is not an option. You can be fined heavily and you can lose your ability to accept credit and debit cards.

One of the twelve requirements is the commitment to monitoring and investigating changes to configuration and password files for any application, server or device involved in the processing of card holder information and transactions. In the case of file content, permissions or attribute changes, logs will only tell me part of the story. Yes a Windows, Linux or Unix log will tell me a file has been changed but it won’t tell me who changed it. It also won’t tell me if the change was authorized or not. To understand who changed a file I need to look at the other user processes running on that server at the same time the file was changed. What user processes were running and who owned them? In Unix or Linux this information is easily viewed with a simple “ps” or “top” command but doesn’t exist in any log. In order to understand if the change was authorized or not I need to compare the log and file change information with the user information and any tickets from the service desk authorizing this user to make this type of modification.

The real reason I believe we need to move on from talking about log management is log management isn’t a market. It isn’t a solution. It is a feature in a much broader landscape of harnessing all the data being generated by our IT infrastructures.

Turning all that data info information for every stakeholder is important to the future of IT as environments grow more complex, dynamic, service oriented, virtualized and mission critical. Not just to report on compliance controls, but to improve our speed of root cause analysis, increase our ability to quickly and comprehensively investigate security attacks and develop more intimate relationships with our customers by better understand their behavior and providing a transparent view of the services they are receiving in return.

The US economy is heading into a recession and technology spending is in for a steep decline in 2008. So every major prognosticator and news outlet from the Wall Street Journal to the Financial Times would have us believe.

Are these people watching the same movie I am? There are two problems I have with this economic hyperbole. Yes that’s what it is. I guess it sells newspapers and gets people to watch things like CNBC. But boy is it misleading.

First of all, in macroeconomics, a recession is a decline in any country’s gross domestic product (GDP), or negative real economic growth, for two or more successive quarters of a year. Yet nobody that I’ve read is forecasting negative growth. They’re forecasting a potential slow down in growth from the current 3.5% per quarter to 1.5 to 2.5% per quarter. But the news outlets feel compelled to use the “R” word just to get attention. Totally irresponsible.

On to my second gripe. With regards to technology and IT spending, I believe, based on what I see, we are in beginning of a long-term gradual increase in IT spending within large enterprises that started eighteen to twenty four months ago.

Sure the current credit crisis may have a short-term impact on budgets within Financial Services companies, but I don’t see any slow down yet. The major consumer, commercial and investment banks we work with have so many critical, revenue generating IT projects in backlog I fail to see how spending is going to slow at all. The telecommunication sector is finally back on the mend after the post early 2000’s bubble and hangover.

Social media, online shopping and the always on dimension of the Internet have online services and large Internet sites like MySpace and Amazon accelerating software, hardware and services spending just to keep up. And security, privacy and compliance initiatives and mandates have companies, service providers and government agencies increasing spending on these items by some 20% or more in 2008 to try and limit their exposure and risk.

Just a month ago the Financial Times had a great piece entitled “What’s on CIO wishlists?” Here’s a quick summary.

1. Business alignment and strategy
2. Hiring and retaining the best staff
3. IT innovation/new methodologies
4. Security
5. Collaboration technologies
6. Controlling costs
7. Compliance and regulation
8. Virtualisation
9. Customer service
10. Mobility (Green issues came 11th)

Doesn’t look like a slow down to me.