Splunk WiT Presents Operationalizing Machine Learning
The Splunk Women in Technology group hosted its first international event in Vancouver on May 24th with fantastic attendance at the British Columbia Technology Industry Association (BCTIA). The event provided the opportunity for technology professionals and students to hear from Splunk experts on Splunk and machine learning.
Carola Thompson, Chief Experience Officer at Splunk, kicked off the event with a brief history of the company and introduced Splunk Research Engineer Dr. Lilian Wong.
Dr. Wong, comes to Splunk with a PhD in Mathematics from Caltech and a passion for computer science. She provided an engaging overview of machine learning and how Splunk can easily perform a wide variety of tasks, including outlier detection, predictive analytics, event clustering, and anomaly detection.
So what is machine learning? Arthur Samuel defined machine learning as a “Field of study that gives computers the ability to learn without being explicitly programmed”. It allows for generalized learning from examples or data. Human brains are good at learning and detecting patterns, but we get lost in volume and details. Computers are great at volume and this is where machine learning comes in. Machine learning is already used in many familiar applications such as spam filtering of your email and weather forecasting, where predictions are made about likely conditions in the future.
There are three common types of machine learning; supervised learning where there is a general set of techniques inferring a model from a set observations based on labels that are fed into the system. Unsupervised learning, when the system is tasked with inferring a model without having access to a set of labeled examples. And finally reinforcement learning where the system is rewarded (or punished) based on the outcomes it generates.
There are numerous machine learning use cases. Dr. Wong highlighted three. For example, within IT Operations, there is predictive maintenance, which involves getting relevant resource usage data, exploring the data and generating predictive models based on past and real time data. Then you apply and validate those models until they are accurate, forecasting resource saturation, demand and usage and finally surfacing incidents to IT Operators. Machine learning can also be applied to security to find insider threats or to business analytics to predict customer churn.
The basic process of machine learning follows a five-step workflow:
- Get all data relevant to the problem
- Explore the data, and fit predictive models on past and real-time data
- Apply and validate models until the predictions are accurate
- Forecast KPIs and metrics associated to the use case
- Surface incidents to an operator, who investigates and acts
Within Splunk, machine learning is used extensively in Splunk User Behavior Analytics, for areas such as behavior baselining and modeling, anomaly detection and advanced threat detection. It is used within the Splunk IT Service Intelligence product for adaptive thresholding and anomaly detection. Finally there is the Splunk machine learning toolkit and showcase for building machine learning applications.
Closing the discussion Dr. Wong walked the audience through building an machine learning application. Firstly, she explained you must get the data, it can be any data, anytime, anywhere for anyone. Then you must explore the data using Splunk search and dashboards. Once your data is explored you can fit and apply models to the data, validating the outcomes until they are accurate. The models are then used to predict how events and operators can act on those predictions. Finally, Dr. Wong explained how operationalizing the models closes the loop of the machine learning process. When you deliver the outcome, she encouraged the audience to keep track of the response and operationalize, repeat the process with a lot of hard work but with the confidence that a lot of great value will come out! She encouraged everyone to try machine learning with the Splunk toolkit and showcase and learn the basics of machine learning further by exploring the textbook Machine Learning (by Murphy).
Machine learning is a challenging subject but you don’t have to be a genius to learn it!