Custom Message Handling and HEC Timestamps with the Kafka Modular Input

Custom Message Handling

If you are a follower of any of my Modular Inputs on Splunkbase , you may see that I employ a similar design pattern across all of my offerings. That being the ability to declaratively plug in your own parameterizable custom message handler to act upon the raw received data in some manner before it gets output to Splunk for indexing. This affords many benefits :

  • Many of my Modular Inputs are very cross cutting in terms of the numerous potential types and formats of data they will encounter once they are let loose in the wild. I can’t think of every data scenario. An extensibility design allows the user and community to be able to customize the
» Continue reading

New support for authoring modular inputs in Node.js

Modular inputs allow you to teach Splunk Enterprise new ways to pull in events from internal systems, third party APIs or even devices. Modular Inputs extend Splunk Enterprise and are deployed on the Splunk Enterprise instance or on a forwarder.  In version 1.4.0 of the Splunk SDK for JavaScript we added support for creating modular inputs in Node.js!

In this post, I’ll show you how to create a modular input with Node.js that pulls commit data from GitHub into Splunk.

Why Node.js

Node.js is designed for I/O intensive workloads. It offers great support for streaming data into and out of a Node application in an asynchronous manner. It also has great support for JSON out of the box. Finally, Node.js has …

» Continue reading

Splunking web-pages

Have you ever had a situation where you found information on a webpage that you wanted to get into Splunk? I recently did and I wrote a free Splunk app called Website Input that makes it easy for everyone to extract information from web-pages and get it into a Splunk instance.

The Problem

There are many cases where web-pages include data that would be useful in Splunk but there is no API to get it. In my case, I needed to diagnose some networking problems that I suspected was related to my DSL connection. My modem has lots of details about the state of the connection but only within the web interface. It supports a syslog feed but it doesn’t include …

» Continue reading

Splunking Social Media: Tracking Tweets


So you use Twitter and have heard Splunk can do “Big Data”. By tapping into Twitter’s API you can use Splunk to investigate the stream of tweets being generated across the globe.

The great thing about using Splunk to do this is that you have complete control of the data meaning it’s incredibly flexible as to what you can build. A few basic ideas I’ve had include tracking hashtags, following specific influencers, or tracking tweets by location in real-time.

What’s more, it takes a matter of minutes before you can start analysing the wealth of data being generated. This post will show you how.…

» Continue reading

Splunking the World Cup 2014: Real Time Match Analysis


As an Englishman I’ve been waiting months – with very high expectations – for the World Cup to come around. Reading fellow Splunker, Matt Davies’ blog post titled, “Splunking World Cup 2014. The winner will be…“, only heightened my excitement.

The tournament is now going into the second week and I’ve been starting to look at the teams, players, and tournament more closely. Which stadium holds the most people? Who’s the top scorer? Which referee hands out the most cards?

With these questions fresh in my mind I opened up Splunk and began to have a look at the huge amounts of information being streamed from the tournament. For this post I’m going to explore real-time match updates; including teams, …

» Continue reading

The Splunk SDK for Python gets modular input support

Support for modular inputs in Splunk 5.0 and later enables you to add new types of inputs to Splunk that are treated as native Splunk inputs.

Last week Jon announced updates to the Splunk SDKs for Java, Python, and JavaScript, now we’ll take a deep dive into modular input support for the Splunk SDK for Python.

The latest release of the Splunk SDK for Python brings modular input support. The Splunk SDKs for C# (see Developing Modular Inputs in C#) and Java also have this functionality as of version and 1.2, respectively. The Splunk SDK for Python enables you to use Python to create new modular inputs for Splunk.

Getting started

The Splunk SDK for Python comes …

» Continue reading

Making SNMP Simpler


From Wikipedia :

Simple Network Management Protocol (SNMP) is an “Internet-standard protocol for managing devices on IP networks”. Devices that typically support SNMP include routers, switches, servers, workstations, printers, modem racks, and more.

SNMP exposes management data in the form of variables on the managed systems.

The variables accessible via SNMP are organized in hierarchies. These hierarchies, and other metadata (such as type and description of the variable), are described by Management Information Bases (MIBs).

MIBs describe the structure of the management data of a device subsystem; they use a hierarchical namespace containing object identifiers (OID). Each OID identifies a variable that can be read or set via SNMP. MIBs use the notation defined by ASN.1.

SNMP agents can …

» Continue reading

Command Modular Input

Simplifying the status quo

I’m often thinking about potential sources of data for Splunk and how to facilitate getting this data into Splunk in the simplest manner possible.

And what better source of data than existing programs on your operating system that already do the heavy lifting for you.

Now this is nothing new to Splunk , we’ve always been able to wrap up a program in a scripted input, execute it, transform the output and pipe it into Splunk.

But rather than going and creating many of these specific program wrappers for Splunk each time you need to capture a program’s output , why not create 1 single Modular Input that can be used as a generic wrapper for …

» Continue reading

Getting data from your REST APIs into Splunk


More and more products,services and platforms these days are exposing their data and functionality via RESTful APIs.

REST really has emerged over previous architectural approaches as the defacto standard for building and exposing web APIs to enable third partys to hook into your data and functionality. It is simple , lightweight , platform independent,language interoperable and re-uses HTTP constructs. All good gravy. And of course , Splunk has it’s own REST API also.

The Data Potential

I see a world of data out there available via REST that can be brought into Splunk, correlated and enriched against your existing data, or used for entirely new uses cases that you might conceive of once you see what is available and …

» Continue reading

Developing Modular Inputs in C#: Part 2

I'm annoyed at our engineering team, but I'll get over it. You see, just hours after I posted my first blog post on writing modular inputs in C#, the team up in Seattle released the latest edition of the C# SDK. Within that SDK is a bunch of class libraries that do a much better job than my work on the scaffolding needed to produce a modular input. I highly recommend you go over to their site and dig in to this. Within this blog post, I'm going to adjust my code to use the new scaffolding and take a look at actually running the code.
» Continue reading