A Way of Thinking about Big Data and Security

I often get asked questions like, “I like Splunk but how much data should I be collecting for security purposes? Is there such a thing as too much data? How do I know what matters in my data?

These are good questions but unfortunately the answer really can be, “it depends.” I still believe there’s no such thing as too much data for security purposes if you are using Splunk. For me there are only two types of data, the data your are using for security and the data you’ll need later that you didn’t think you needed at the time. There will come a time when security folk will be looking at the fidelity of the data as an indicator of ‘the unusual.’ I’ve also always believe that collected data doesn’t have meaning unless I could use it to figure out how to use it to show changes in confidentiality, integrity and/or availability that might put the business at risk.

Lately I’ve begun to get a bit uncomfortable with the ‘it depends’ answer I’ve been giving and I started to search for a process I could use to provide better answers. Help with this came to me out of the blue in the form of an article in CRM Magazine (yes CRM Magazine) called Big Data Analytics Can Help Improve Information Security. In it, the author interviewed Steve Durban with the International Security Forum. What caught my eye was at the end of the article where Steve outlined what he called, “The ‘Human Element’ of the Big Data Equation. He went on to outline five steps organizations can take to analyze big data for information security purposes.

  • Identify the business issue;
  • Construct a hypothesis to be tested;
  • Select the relevant data sources and provide subject matter expertise about them;
  • Determine the analyses to be performed; and
  • Interpret the results.

I took these steps and put them into an illustration that makes it easier on the eyes and added a sixth step which is to iterate on the results. I also added an example of this type of thinking below.

Using Big Data for Security -- A Process

Using the Big Data for Security Process

This approach links data types and amounts to business risk requirements and helps us examine problems through the lens of confidentiality, integrity and availability.