Detect IoT anomalies and geospatial patterns for logistics insights

In part 1 of this blog series we spoke about how to turn sensor data into logistics insights. In this part we outline one approach for anomaly detection and enrich our sensor data with location information to discover geospatial patterns.

Anomalies? Find them with a few lines of SPL.

Anomaly detection can be tricky and implementations vary from simple thresholding and baselining to highly sophisticated approaches based on machine learning. In this example we leveraged the Splunk Machine Learning Toolkit to detect numeric outliers using a sliding window approach to check against multiples of the standard deviation in this time series to spot anomalies.

iot_blog_timechart_anomalies_chart

And that’s how the SPL looks like:

index="sensor"
| timechart span=1s avg(ax) as avx avg(ay) as

» Continue reading

Turn IoT sensor data into Operational Intelligence for logistics

The Internet of Things (IoT) wave may impact businesses and industry verticals differently but with the same potential: IoT opens new doors to interesting use cases that have immediate business impact and value. Splunk has delivered Operational Intelligence and Analytics in IT and Security for years, so why not apply Operational Intelligence and Analytics to IoT?

IoT_logistics_overviewReferring to the general definition of IoT we consider an object that is connected to the internet, in our case data coming from a sensor which measures acceleration. One use case I want to walk through here is not new to logistics, but a great example to show the value in IoT. As the diagram above depicts the globalized delivery of goods takes place …

» Continue reading

Dashboard Digest Series – Episode 2

noaa_website

Welcome to the second episode of the Dashboard Digest Series! So what do we have for Episode 2? Waves!

The use case here was to display real-time and historical parameters and statistics from the National Oceanic and Atmospheric Administrations National Data Buoy Center or NOAA NDBC for short.  Thanks to an add-on created by Julien Ruaux on Splunkbase, I was able to easily collect data from the NDBC’s data feed and start creating dashboards right away.   While the NOAA NDBC site has it’s own dashboard (pictured right) I figured it might be useful to access and visualize the data in different ways through Splunk.  That and eventually correlate the buoy data with other data sources.

Purpose: Display meaningful statistics …

» Continue reading

Talk to Splunk with Amazon Alexa

What do you think the future experience of interacting with your data is going to be like ? Is it going to be logging in by way of a user interface and then using your mouse/keyboard/gestures to view and interact with something on a display panel , or is it going to be more like simply talking with another person ?

Introducing the “Talk to Splunk with Amazon Alexa” App

This is a Splunk App that enables your Splunk instance for interfacing with Amazon Alexa by way of a custom Alexa skill, thereby provisioning a Natural Language interface for Splunk.

You can then use an Alexa device such as Amazon’s Echo,Tap or Dot or another 3rd party hardware device to tell …

» Continue reading

Splunk at ThingMonk 2016

ThingmonkHi everyone

I’m Duncan Turnbull and I am the technical lead for the Analytics and IoT practice team here at Splunk in Europe. This means I get to spend my time listening, explaining, showing and talking to organizations across EMEA about how to use their machine data to solve business problems and find the value from it by using Splunk’s software.

I’m delighted to be at Redmonk’s ThingMonk event this year at the Hack Day on day 0. I’ll be there to see what we can build on the day, build some cool things myself and showcase how to use all the data from these sensors. Last year we had Matt Davies and James Hodge from Splunk present and …

» Continue reading

#splunkconf16 preview: IT Operations Track – Choose your own adventure!

Does anyone else remember the ‘choose your own adventure books’ from the 90s? I do, and this year’s #splunkconf16 has me almost as excited as getting a brand spankin’ new pile of books. Just kidding, 2016 user conference is going to be much, much better!

2016-05-09-1462761733-5966723-chooseyourown

caveoftime

(No, this is not an ITSI Glass Table)

 

Splunk .conf2016 is coming up fast, and everyone on the Splunk team is excited to head down to the happiest place on earth for this year’s user conference. Check out some key details below about the great sessions that will be featured in the Splunk IT Operations track this year at .conf 2016. This year, we’ve made it easy for you by parsing the sessions into …

» Continue reading

Adding a Deployment Server / Forwarder Management to a new or existing Splunk Cloud (or Splunk Enterprise) Deployment

As part of the Cloud Adoption team, I am working with Splunk Cloud (and Splunk Enterprise) customers on a daily basis and I get asked questions quite frequently about how to optimize, and effectively reduce, administration overhead. This becomes especially relevant when I am talking with new or relatively new customers that are expanding from a handful of forwarders, into the 100’s or 1000’s of forwarders. And I always say…. start with a Deployment Server.

For larger customers that have trained and experienced Splunk Administrators, or have engaged with Professional Services, this is a given and typically already exists in their deployments.

On the other end however, new Splunk Cloud and Splunk Enterprise customers may not have this luxury.…

» Continue reading

Send data to Splunk via an authenticated TCP Input

Wow , my second blog in 24 hrs about Protocol Data Inputs(PDI) , but sometimes you just infected with ideas and have to roll with it.

So my latest headbump is about sending text or binary data to Splunk over raw TCP and authenticating access to that TCP input.Simple to accomplish with PDI.

Setup a PDI stanza to listen for TCP requests

PDI has many options , but for this simple example you only need to choose the protocol(TCP) and a port number.


Screen Shot 2016-07-30 at 3.31.08 PM

Declare a custom handler to authenticate the received data

You can see this above in the Custom Data Handler section.I have declared the handler and  the authentication token that the handler should use via a JSON properties …

» Continue reading

Sending binary data to Splunk and preprocessing it

A while ago I released an App on Splunkbase called Protocol Data Inputs (PDI)  that allows you to send text or binary data to Splunk via many different protocols and dynamically apply pre processors to act on this data prior to indexing in Splunk. You can read more about it here.

I thought I’d just share this interesting use case that I was fiddling around with today. What if I wanted to send compressed data (which is a binary payload) to Splunk and index it ? Well , this is very trivial to accomplish with PDI.

Choose your protocol and binary data payload

PDI supports many different protocols , but for the purposes of this example I just rolled  a …

» Continue reading

True Machine Learning is finally here for all to Leverage

In this short post I want to hit 3 simple points.

  • Why has true machine learning been so difficult to provide to the masses?
  • Why is machine learning not simply statistical models?
  • What type of organization has the power to bring machine learning to the masses?

The simple reason it took so long to bring machine learning from the theoretical to the everyday is that it is hard. No scratch that it is really – really hard to do at scale and price point where every organization can leverage the innate power of machine learning.

Think about it, you have this layer of intelligence over all your machine data that is constantly on the watch for unusual behavior and anomalies. …

» Continue reading