Using machine learning for anomaly detection research

Over the last years I had many discussions around anomaly detection in Splunk. So it was really great to hear about a thesis dedicated to this topic and I think it’s worth sharing with the wider community. Thanks to its author Niklas Netz in advance!

Obviously anomaly detection is an important topic in all core use case areas of Splunk, but each one has different requirements and data, so unfortunately there is not always an easy button. In IT Operations you want to detect systems outages before they actually occur and proactively keep your depending services up and running to meet your business needs. In Security you want to detect anomalous behavior of entities to detect potential indicators for breaches …

» Continue reading

Analyzing BotNets with Suricata & Machine Learning

Since the official rollout at the year’s. conf of the Machine Learning Toolkit(MLTK), Splunkers have been pursing some interesting use cases ranging from IT operations, planning, security and business analytics. Those use cases barely scratch the surface of what is possible with machine learning and Splunk. As an example, I will use the machine learning toolkit and data collected from Suricata to analyze botnet populations. This population analysis will be used to create a model for predicting the Mirai botnet based on network features.

Suricata

Suricata is an open source threat detection engine, which can be run in passive mode for intrusion detection or inline for intrusion prevention. My lab environment is configured for intrusion detection, meaning Suricata will not …

» Continue reading

What is your “Art of the Possible” Idea?

Screen Shot 2016-11-03 at 2.45.28 PM

Allow me to paint a quick picture for you and then ask a few simple questions that are intended to significantly advance your career.

You work at a company, in an organization and at a job that pays the bills.  Your job is two parts keep the train on the tracks and one part emergency repair person.  Said another way, the long bouts of mundane routine is interrupted by emergencies not of your doing and most of the time not your responsibility but hey, when something breaks everyone gets involved.

The above story could be someone in IT, Security – or even the business (as we all have our part to do and each person contributes to the daily function of …

» Continue reading

Detect IoT anomalies and geospatial patterns for logistics insights

In part 1 of this blog series we spoke about how to turn sensor data into logistics insights. In this part we outline one approach for anomaly detection and enrich our sensor data with location information to discover geospatial patterns.

Anomalies? Find them with a few lines of SPL.

Anomaly detection can be tricky and implementations vary from simple thresholding and baselining to highly sophisticated approaches based on machine learning. In this example we leveraged the Splunk Machine Learning Toolkit to detect numeric outliers using a sliding window approach to check against multiples of the standard deviation in this time series to spot anomalies.

iot_blog_timechart_anomalies_chart

And that’s how the SPL looks like:

index="sensor"
| timechart span=1s avg(ax) as avx avg(ay) as

» Continue reading

Turn IoT sensor data into Operational Intelligence for logistics

The Internet of Things (IoT) wave may impact businesses and industry verticals differently but with the same potential: IoT opens new doors to interesting use cases that have immediate business impact and value. Splunk has delivered Operational Intelligence and Analytics in IT and Security for years, so why not apply Operational Intelligence and Analytics to IoT?

IoT_logistics_overviewReferring to the general definition of IoT we consider an object that is connected to the internet, in our case data coming from a sensor which measures acceleration. One use case I want to walk through here is not new to logistics, but a great example to show the value in IoT. As the diagram above depicts the globalized delivery of goods takes place …

» Continue reading

#splunkconf16 preview: What’s the next big thing in big data? Machine learning.

SCL-Splunk-conf2016-preview-BigDataIdeas_twtr1-440x220Big data, especially machine data, is fueling the latest machine learning (ML) trend and we’ve got you covered with 18 sessions at .conf2016. Cut through the hype and learn how to operationalize ML in your organization to prevent service outages, manage inventory, identify insider threats, or to simply manage your alerts better. Whether you’ve been using the ML Toolkit since it was introduced last year or you’re just curious what all the excitement is about, you can hear directly from Splunk product managers and developers and customers like Emerson, NTT Docomo, Dunkin’ Donuts, Zillow, and others.

Tuesday, September 27:

» Continue reading

True Machine Learning is finally here for all to Leverage

In this short post I want to hit 3 simple points.

  • Why has true machine learning been so difficult to provide to the masses?
  • Why is machine learning not simply statistical models?
  • What type of organization has the power to bring machine learning to the masses?

The simple reason it took so long to bring machine learning from the theoretical to the everyday is that it is hard. No scratch that it is really – really hard to do at scale and price point where every organization can leverage the innate power of machine learning.

Think about it, you have this layer of intelligence over all your machine data that is constantly on the watch for unusual behavior and anomalies. …

» Continue reading

Top Technical Questions on Splunk UBA

With the acquisition of Caspida (now Splunk UBA) in July of 2015, we have been talking to many customers regarding user and entity behavioral analytics. Our customers have been asking questions about how this type of threat detection product works, and in this blog, I’m going to discuss some of the most common questions, along with answers and/or explanations from a security researcher and practitioner’s viewpoint.

 

What makes Splunk UBA unique compared to detection technologies?

Splunk UBA uses an unsupervised machine-learning based approach to determine whether events generated from multiple data sources are anomalies and/or threats. This is a turnkey approach that does not require customers to train the models, and does not require administrators to develop signatures in …

» Continue reading

Security Solutions Need Data Science and Machine Learning to Protect Organizations

Screen Shot 2015-11-03 at 3.42.46 PMEvery month we hear about a major breach targeting an enterprise or public sector. Based on current cyberattack growth rates, we anticipate the impact to our global economy to be around three trillion US dollars.

Within the past five years, 2.5 billion records were exposed. From January, 2015 until June 2015, 256 million records were compromised. Breaking that down, that’s…

  • 1,400,000 stolen records per day (or)
  • 56,000 stolen records per hour (or)
  • 943 stolen records per minute.

A recent FireEye study found that on average, an organization takes 205 days to detect advanced threats. We need a security solution that uses a new paradigm to combat modern day attacks…

Splunk calls it Splunk User Behavior Analytics (Splunk UBA).

Splunk UBA

» Continue reading

Splunk Acquires Caspida: The Future in Advanced Breach Detection is Here

logo-dark

Today, we welcome Caspida to the Splunk family. This acquisition enables Splunk to bring critical analytical capabilities to our customers and extends Splunk’s security analytics leadership. Caspida adds data science-driven Behavioral Analytics to the industry’s most powerful analytics-enabled SIEM solution.

In the last year, I have had several conversations with peers and customers about attack patterns and enterprise compromises. We see three big categories of attackers:

  • Advanced or nation state attackers: they compromise, persist, and run campaigns – not just one off opportunistic attacks.
  • Insiders: trusted parties that abuse their privileges.
  • Fraudsters or cyber criminals: stealing money, credit cards, estore wallets, and conduct fraudulent transactions like wire transfers, and reimbursement or benefits fraud.

All recent high-profile …

» Continue reading