From API to easy street within minutes

30? 20? …15? It all depends on how well you know your third-party API. The point is that polling data from third-party APIs is easier than ever. CIM mapping is now a fun experience.

Want to find out more about what I mean?  Read the rest of this blog and explore what’s new in Add-on Builder 2.1.0.

REST Connect… and with checkpointing

Interestingly  this blog happens to address a problem I faced back on my very first project at Splunk. When I first started at Splunk as a Sales engineer, I  worked on  building a prototype of the ServiceNow Add-on. Writing Python, scripted inputs vs mod input, conf files, setup.xml, packaging, best practices, password encryption, proxy and even checkpointing… the list goes …

» Continue reading

Using machine learning for anomaly detection research

Over the last years I had many discussions around anomaly detection in Splunk. So it was really great to hear about a thesis dedicated to this topic and I think it’s worth sharing with the wider community. Thanks to its author Niklas Netz in advance!

Obviously anomaly detection is an important topic in all core use case areas of Splunk, but each one has different requirements and data, so unfortunately there is not always an easy button. In IT Operations you want to detect systems outages before they actually occur and proactively keep your depending services up and running to meet your business needs. In Security you want to detect anomalous behavior of entities to detect potential indicators for breaches …

» Continue reading

Data-driven insights into performance, availability and compliance of an FX service with ITSI

Running a Service in financial services is tough.  Not only is there the burden of ensuring your service is 100% available to avoid a financial meltdown but you also need to find ways of optimizing every available CPU clock or KB of memory to stay competitive (check out long-time Splunker Finnbar Cunningham’s Credit Suisse .conf Presentation). All whilst having regulators breathing down your neck ready to slap a multi-million (or billion) dollar fine on you.

Maybe you do deserve those bonuses.

What do I mean by a Service? A system or application that allows a business to operate or trade.  This could be an online retail site, an ERP platform and so on.  For financial services, and capital markets in …

» Continue reading

How to Stop Playing the Blame Game in Your IT Department

It’s a familiar scenario: a problem is discovered, and a Service Desk Team gets a help ticket. The Service Desk Team tells Operations that there’s an outage. The Operations Team suggests that the problem could be the result of bad code and passes the issue to Dev. The Dev Team responds that it doesn’t have the tools to solve the problem and asks for logs from production systems.

Suddenly the situation is escalated.

A war room’s assembled. Here you’ll often find a DBA, Docker specialist, network specialist, release manager, site reliability engineer and a developer, sometimes calling in remotely from separate locations. The pressure’s on for everyone to prove their innocence and confirm individual components of the infrastructure are ok. …

» Continue reading

It’s cold outside – A year’s worth of data from my pellet heating unit!

Hello,

Over a year ago I shared a blog about my efforts to monitor the pellet unit I use for heating my home. I wanted to share a report on how it went with access to a year’s worth of data and what new tuning steps I have added as a result. Remember – I’m an IT guy – not a pellet home heating system expert – but as I spoke to the experts they got really excited about what I found and how I could fine tune everything – so I thought I would share it with you so that you too can mess up the configuration of your home heater as well ;-).

A year’s worth of data

» Continue reading

Analyzing BotNets with Suricata & Machine Learning

Since the official rollout at the year’s. conf of the Machine Learning Toolkit(MLTK), Splunkers have been pursing some interesting use cases ranging from IT operations, planning, security and business analytics. Those use cases barely scratch the surface of what is possible with machine learning and Splunk. As an example, I will use the machine learning toolkit and data collected from Suricata to analyze botnet populations. This population analysis will be used to create a model for predicting the Mirai botnet based on network features.

Suricata

Suricata is an open source threat detection engine, which can be run in passive mode for intrusion detection or inline for intrusion prevention. My lab environment is configured for intrusion detection, meaning Suricata will not …

» Continue reading

Improving Visibility in Security Operations with Search-Driven Lookups

Looking back on 2016, Splunk Enterprise Security added significant capabilities to its platform for security operations, including Adaptive Response, User & Entity Behavior Analytics (UEBA) integration and Glass Tables.  Another capability that was added, but has received less attention is a new type of search that Splunk calls Search-Driven Lookups.  Because there has not been as much attention put to this, I wanted to share a bit about this capability and how it can be used.

Search-Driven Lookups originated from a question that users of legacy SIEM providers often asked; how can Splunk dynamically create watchlists that can then be used in correlating new events against a watchlist?  Enterprise Security has had the ability to correlate against a …

» Continue reading

Dashboard Digest Series – Episode 5: Maps!

splunk_maps“A map does not just chart, it unlocks and formulates meaning; it forms bridges between here and there, between disparate ideas that we did not know were previously connected.” ― Reif Larsen, The Selected Works of T.S. Spivet

Welcome to Episode 5 of the Dashboard Digest series!

Maps play a critical role in visualizing machine data in almost any industry for thousands of use cases.  We’ve been continuously adding more mapping functionality to Splunk and with the recent addition of Custom Visualizations in Splunk 6.4 you (the community) have too!  This is exciting news as I’ve noticed many times the first panel on a dashboard that draws attention is a map.  The best part is that each of these displays …

» Continue reading

Gaze into Splunk’s Crystal Ball for What’s to Come in 2017

social-splunk-2017predictionsLast year, a team of Splunkers came up with several predictions for what 2016 would bring in the fields of IT, security, and big data. This year we’ve done it again, looking into our crystal ball (or industry experience) to share our prophecies for 2017.

But first, let’s look back at some of the hits and misses of what we predicted for 2016.

Behavioral analysis will shift from an emphasis on user credentials to machine-to-machine credentials.

Haiyan Song, our SVP of security markets, predicted that “anomaly detection will become less about analyzing users or entities and more about leveraging machine learning and data science.” While there’s still a way to go, this has begun to come true: As

» Continue reading

Splunk App for Jenkins: Increase quality and velocity of your software releases

jenkins-stickersMore than 12,000 customers are using Splunk software to monitor their critical services. They are deploying Splunk software on-premises or using Splunk Cloud, across a variety of operating systems, different deployment configurations and scale complexities. As a result, ensuring the highest Splunk software quality is of paramount importance.

To help with this endeavor, our engineering team is running most of our development and testing workflows using the Jenkins platform. The complexity of developing and testing Splunk software across various configurations has resulted in a massive Jenkins deployment infrastructure with multiple clusters, with more than 500 slaves per master running thousands of jobs per day. Previously, analyzing Jenkins data at this scale manually was not possible. As a result, our …

» Continue reading