This following blog post has turned into more than just a post. It’s more of a paper. In any case, in the post I am trying to capture a number of concepts that are defining the IT data management market.
When I am talking about IT data management, I am talking about the over-arching market that covers anything from log management to security information management and security event management.
Any company or IT department/operations can be placed along the maturity scale (see Figure 1). The further on the right, the more mature the operations with regards to IT data management. A company generally moves along the scale. A movement to the right does not just involve the purchase of new solutions or tools, but also needs to come with a new set of processes. Products are often necessary but are not a must.
The further one moves to the right, the fewer companies or IT operations can be found operating at that scale. Also note that the products that companies use are called log management tools for the ones located on the left side of the scale. In the middle, it is the security information and event management (SIEM) products that are being used, and on the right side, companies have to look at either in-house tools, scripts, or in some cases commercial tools in markets other than the security market. Some SIEM tools are offering basic advanced analytics capabilities, but they are very rudimentary. The reason why there are no security specific tools and products on the right side becomes clear when we understand a bit better what the scale encodes.

Figure 1: IT Data Management Maturity Scale.

The Maturity Scale

Let us have a quick look at each of the stages on the scale. (Skip over this if you are interested in the conclusions and not the details of the scale.)

• Do nothing: I didn’t even explicitly place this stage on the scale. However, there are a great many companies out there that do exactly this. They don’t collect data at all.
• Collecting logs: At this stage of the scale, companies are collecting some data from a few data sources for retention purposes. Sometimes compliance is the driver for this. You will mostly find things like authentication logs or maybe message logs (such as email transaction logs or proxy logs). The number of different data sources is generally very small. In addition, you mostly find log files here. No more specific IT data, such as multi-line applications logs or configurations.
• Forensics / Troubleshooting: While companies in the previous stage simply collect logs for retention purposes, companies in this stage actually make use of the data. In the security arena they are conducting forensic investigations after something suspicious was noticed or a breach was reported. In IT operations, the use-case is troubleshooting. Take email logs, for example. A user wants to know why he did not receive a specific email. Was it eaten by the SPAM filter or is something else wrong?
• Save searches: I don’t have a better name for this. In the simplest case, someone saves the search expression used with a grep command. In other cases, where a log management solution is used, users are saving their searches. At this stage, analysts can re-use their searches at a later point in time to find the same type of problems again, without having to reconstruct the searches every single time.
• Share searches: If a search is good for one analyst, it might be good for another one as well. Analysts at some point start sharing their ways of identifying a certain threat or analyze a specific IT problem. This greatly improves productivity.
• Reporting: Analysts need reports. They need reports to communicate findings to management. Sometimes they need reports to communicate among each other or to communicate with other teams. Generally, the reporting capabilities of log management solutions are fairly limited. They are extended in the SEM products.
• Alerting: This capability lives in somewhat of a gray-zone. Some log management solutions provide basic alerting, but generally, you will find this capability in a SEM. Alerting is used to automate some of the manual trouble-shooting that is done among companies on the left side of the scale. Instead of waiting for a user to complain that there is something wrong with his machine and then looking through the log files, analysts are setting up alerts that will notify them as soon as there are known signs of failures showing up. Things like monitoring free disk space are use-cases that are automated at this point. This can safe a lot of manual labor and help drive IT towards a more automated and pro-active discipline.
• Collecting more logs and IT data: More data means more insight, more visibility, broader coverage, and more uses. For some use-cases we now need new data sources. In some cases it’s the more exotic logs, such as multi-line application logs, instant messenger logs, or physical access logs. In addition more IT data is needed: configuration files, host status information, such as open ports or running processes, ticketing information, etc. These new data sources enable a new and broader set of use-cases, such as change validation.
• Correlation: The manual analysis of all of these new data sources can get very expensive and too resource intense. This is where SEM solutions can help automate a lot of the analysis. Uses like correlating trouble tickets with file changes, or correlating IDS data with operating system logs (Note that I didn’t say IDS and firewall logs!) There is much much more to correlation, but that’s for another blog post.

Note the big gap between the last step and this one. It takes a lot for an organization to cross this chasm. Also note that the individual mile-stones on the right side are drawn fairly close to each other. In reality, think of this as a log scale. These mile-stones can be very very far apart. The distance here is not telling anymore.

• Visual analysis: It is not very efficient to read through thousands of log messages and figure out trends or patterns, or even understand what the log entries are communicating. Visual analysis takes the textual information and packages them in an image that conveys the contents of the logs. For more information on the topic of security visualization see Applied Security Visualization.
• Pattern detection: One could view this as advanced correlation. One wants to know about patterns. Is it normal that when the DNS server is doing a zone transfer that you will also find a number of IDS alerts along with some firewall log entries? If a user browses the Web, what is the pattern of log files that are normally seen? Patter detection is the first step towards understanding an IT environment. The next step is to then figure out when something is an outlier and not part of a normal pattern. Note that this is not as simple as it sounds. There are various levels of maturity needed before this can happen. Just because something is different does not mean that it’s a “bad” anomaly or an outlier. Pattern detection engines need a lot of care and training.
• Interactive visualization: Earlier we talked about simple, static visualization to better understand our IT data. The next step in the application of visualization is interactive visualization. This type of visualization follows the principle of: “overview first, zoom and filter, then details on demand.” This type of visualization along with dynamic queries (the next step) is incredibly important for advanced analysis of IT data.
• Dynamic queries: The next step beyond interactive, single-view visualizations are multiple views of the same data. All of the views are linked together. If you select a property in one graph, the selection propagates to the others. This is also called dynamic queries. This is the gist of fast and efficient analysis of your IT data.
• Anomaly detection: Various products are trying to implement anomaly detection algorithms in order to find outliers, or anomalous behavior in the IT environment. There are many approaches that people are trying to apply. So far, however, none of them had broad success. Anomaly detection as it is known today is best understood for closed use-cases. For example, NBADs are using anomaly detection algorithms to flag interesting findings in network flows. As of today, nobody has successfully applied anomaly detection across heterogeneous data sources.
• Sharing views, patterns, and outliers: The last step on my maturity scale is the sharing of advanced analytic findings. If I know that certain versions of the Bind DNS server tend to trigger a specific set of Snort IDS alerts, it is something that others should know as well. Why not share it? Unfortunately, there are no products that allow us to share this knowledge.

While reading the maturity scale, note the gaps between the different stages. They signify how quickly after the previous step a new step sets in. If you were to look at the scale from a time-perspective, you would start an IT data management project on the left side and slowly move towards the right. Again, the gaps are fairly indicative of the relative time such a project would consume.

Related Quantities

The scale could be overlaid with a lines showing some interesting, related properties. I decided to not do so in favor of legibility. Instead, have a look at Figure 2. It encodes a few properties: number of products on the market, number of customers / users, and number of data sources needed at that state of maturity.

Figure 2: The number of product, companies, and data sources tat are used / available along the maturity scale.

Why are so few products on the right side of the scale? The most obvious reason is one of market size. There are not many companies on the right side. Hence there are not many products. It is sort of a chicken and an egg problem. If there were more products, there might be more companies using them - maybe. However, there are more reasons. One of them being that in order to get to the right side, a company has to traverse the entire scale on the left. This means that the potential market for advanced analytics is the amount of companies that linger just before the advanced analytics market itself. That market is a very small one. The next question would be why there are not more companies close to the advanced analytics stage? There are multiple reasons. Some of them are:

• Not many environments manage to collect enough data to implement advanced analytics across heterogeneous data. Too many environments are stuck with just a few data sources. There are organizational, architectural, political, and technical reasons why this is so.
• A lack of qualified people (engineers, architects, etc) is another reason. Not many companies have the staff that understands how to deal with all the data collected. Not many people understand how to interpret the vast amount of different data sources.

The effects of these phenomenon play yet again into the availability of products for the advanced analytics side of the scale. Because there are not many environments that actually collect a diverse set of IT data, companies (or academia) cannot conduct research on the subject. And if they do, they mostly get it wrong or capture just a very narrow use-case.

What Else Does the Maturity Scale Tell Us?

Let us have a look at some of the other things that we can learn from/should know about the maturity scale:

• What does it mean for a company to be on the far right of the scale?
  • In-depth understanding of the data
  • Understanding of how to apply advanced analytics, such as visualization theory, anomaly detection, etc)

It is the time of the year where everyone publishes their predictions for the upcoming year. In past years, I have refrained from publishing my own predictions. This year I am going to change that and I will take a stab. I don’t have any earth shattering things to say and I am covering quite a broad set of topics. Anyways, maybe you find one or two interesting things:

• Security and IT spending: Security projects have never been the ones that were easy to fund (except right after a big worm outbreak, which we haven’t had in years). With the current economical situation, the security budgets for 2009 are not going to be any easier to justify. Therefore, we will see a convergence of projects. Security is going to piggy-back on other IT projects, for example, change management. CM is an integral part of a lot of security requirements, such as PCI. Visibility into the IT infrastructure is another project that will help fund security. SIM, SEM, SIEM, or ESLIM (no kidding, this exists! It wasn’t me. Blame the 451 group!) will need to extend their messages and capabilities to show how they can help provide visibility into the complete IT environment. IT search is going to be especially well situated for that.
• Security ROI: Calculating an ROI for security is hard. It’s an often discussed topic among security experts. 2009 is not going to give us yet another formula to compute the ROI. However, as mentioned earlier, security will be used as an opportunity to optimize IT. Questions like: “How can you do more with less?” will be used to compute an ROI. A lot of companies have consolidation on their agendas for the new year. 75% of the solutions and tools will be eliminated. The tasks of those tools will have to be covered with the remaining 25%. A great opportunity for security monitoring tools to broaden their footprint.
• Metrics: 2008 was supposed to be the year of risk management. I didn’t feel much of that. Or have you seen a push in risk management products? 2009 is going to be the year of metrics. People have to measure things. Not necessarily pure security metrics, but IT metrics, such as productivity, resources, MTTI, etc. Products will have to show actual, measurable benefits. It’s all about cost and how to reduce it. Without metrics you cannot assess how much a tool helps you safe.
• More visibility: It is amazing, but a lot of companies don’t even know what assets/machines they operate. How can you do anything without that information? And that is just the top of the ice berg. IT needs more visibility. What is running where? How well are things running? How efficient? This plays into Green IT also, where you need to know how well servers are utilized how much power they consume and what the temperature is across the data center. Visibility also includes things like identity management. We need to know who executed a task or committed a transaction. It’s not of much use if we know that a certain machine attacked us. We want to know who is behind the activity. The question in 2009 is going to be how to integrate your asset management and IdM into your monitoring infrastructure.
• Consolidation: We have seen acqusitions happen all through 2008. There will be much more. Just along the lines of the security initiatives being coupled closer and closer with IT initiatives, products/companies will be merging.
• Visualization market/tools: What will be going on in the visualization market? Not too much. People are not ready. A lot of companies are still struggling with centralizing IT data. They are starting to use the data to troubleshoot problems. Beyond that, advanced analytics, such as visualization, are not commonly used yet. On the brighter side, new tools will enter the market. DAVIX will come out with a new release, hopefully early 2009. This will help make visualization available to the broader masses. The new release is going to have Splunk integrated, which should help manage all the IT data! In addition, a slew of new visualization tools will be available in the distro. Hopefully, this will help broaden the security visualization community.
• Interoperability: This is a topic that I am fairly passionate about. I have been doing quite a lot of work on the topic of how to get machines to talk to each other through events, logs, and generic IT data. Recently a new syslog RFC was published. I was much too late to actually comment on it. It has good intentions, but it is definitely not what I would like it to be. CEE is still alive, despite the lack of new publications. 2009 will bring us at least one release of one of the sub-standards. If I had to take a guess it would be the syntax and accompanying dictionary. Well, maybe just the dictionary. And definitely will we start collecting log recommendations. That will happen very soon now!
• No data sets: Over many years, we have been facing a huge problem in the research arena. Nobody has solved it yet. It’s the problem of data sets. Researches need data sets to verify their algorithms and approaches. Guess what, 2009 will not solve this. Unless someone comes up with a really great way of anonymizing data, data sets will not be shared. People are not sharing their logs without being absolutely sure that there is no confidential data leaking. I have a feeling that will we be able to solve this only with cryptography. Something along the lines of secure voting schemes, where the analysis would happen on encrypted data. But how do you do that? I have no idea. Until then, people will keep doing verification and analysis on synthetic, old, and irrelevant data sets.