Using machine learning for anomaly detection research

Over the last years I had many discussions around anomaly detection in Splunk. So it was really great to hear about a thesis dedicated to this topic and I think it’s worth sharing with the wider community. Thanks to its author Niklas Netz in advance!

Obviously anomaly detection is an important topic in all core use case areas of Splunk, but each one has different requirements and data, so unfortunately there is not always an easy button. In IT Operations you want to detect systems outages before they actually occur and proactively keep your depending services up and running to meet your business needs. In Security you want to detect anomalous behavior of entities to detect potential indicators for breaches …

» Continue reading

Kaufland DevSummit2016 – Splunk for DevOps – Faster Insights, better code

The first DevSummit event was recently hosted by Kaufland with 200 people attending for the day to hear presentations about the “World of API”, discuss the latest best practice developments and build ideas in a hackathon. One highlight was the keynote from Markus Andrezak on how technology, business and innovation play together.

Of course, a team of Splunkers (big thanks to my colleagues Mark and Henning) wouldn’t miss such an event and got involved with a booth as well as a presentation. It was amazing to have so many fruitful discussions about how to make data more easily accessible and useable for business, development and operation teams. In the morning Joern Wanke from the Kaufland Omnichannel team presented on how …

» Continue reading

Detect IoT anomalies and geospatial patterns for logistics insights

In part 1 of this blog series we spoke about how to turn sensor data into logistics insights. In this part we outline one approach for anomaly detection and enrich our sensor data with location information to discover geospatial patterns.

Anomalies? Find them with a few lines of SPL.

Anomaly detection can be tricky and implementations vary from simple thresholding and baselining to highly sophisticated approaches based on machine learning. In this example we leveraged the Splunk Machine Learning Toolkit to detect numeric outliers using a sliding window approach to check against multiples of the standard deviation in this time series to spot anomalies.

iot_blog_timechart_anomalies_chart

And that’s how the SPL looks like:

index="sensor"
| timechart span=1s avg(ax) as avx avg(ay) as

» Continue reading

Turn IoT sensor data into Operational Intelligence for logistics

The Internet of Things (IoT) wave may impact businesses and industry verticals differently but with the same potential: IoT opens new doors to interesting use cases that have immediate business impact and value. Splunk has delivered Operational Intelligence and Analytics in IT and Security for years, so why not apply Operational Intelligence and Analytics to IoT?

IoT_logistics_overviewReferring to the general definition of IoT we consider an object that is connected to the internet, in our case data coming from a sensor which measures acceleration. One use case I want to walk through here is not new to logistics, but a great example to show the value in IoT. As the diagram above depicts the globalized delivery of goods takes place …

» Continue reading

Splunk in Space: Splunking Satellite Data in the Cloud

Hello all,

This year a Team of Splunkers attended the ESA App Camp 2015 in lovely Frascati, Italy. The topic of this year’s challenge was:

“There are thousands of ways to enrich apps with data from space – what’s yours?”

The Splunk team featured Robert Fujara and Philipp Drieger alongside with camp participants Claire Crotty and Anthony Thomas. Together the team created a mobile web app that accessed a Splunk Cloud instance to analyze geolocation-based satellite data and inform users about different environmental indicators across Europe. Users can input their preferences in terms of living environment and based on different indicators they then receive recommendations on which city or region would suit them best. 

The key data sources for this project

» Continue reading

All aboard with Infrastructure 4.0 — Splunk wins Deutsche Bahn Internet of Things Hackathon

Deutsche Bahn (DB) describes itself as the second largest transport company in the world and is the largest railway and infrastructure operator in Europe. With the popularity of Industry 4.0 and IoT in Germany, DB recently ran a “Deutsche Bahn goes 4.0” Hackathon over the weekend of May 8-9 2015. The concept was “We provide the data, you innovate with it”. Splunk participated with a crack team of two people, a copy of Splunk Enterprise running on a laptop and got their hands dirty digging into a labyrinth of infrastructure data. The challenge was tough: starting at 5pm we had 24 hours straight to analyze the data and demonstrate the value from it. After the final presentation of …

» Continue reading