Custom Message Handling and HEC Timestamps with the Kafka Modular Input
Custom Message Handling
If you are a follower of any of my Modular Inputs on Splunkbase , you may see that I employ a similar design pattern across all of my offerings. That being the ability to declaratively plug in your own parameterizable custom message handler to act upon the raw received data in some manner before it gets output to Splunk for indexing. This affords many benefits :
- Many of my Modular Inputs are very cross cutting in terms of the numerous potential types and formats of data they will encounter once they are let loose in the wild. I can’t think of every data scenario. An extensibility design allows the user and community to be able to customize the
New support for authoring modular inputs in Node.js
In this post, I’ll show you how to create a modular input with Node.js that pulls commit data from GitHub into Splunk.
Node.js is designed for I/O intensive workloads. It offers great support for streaming data into and out of a Node application in an asynchronous manner. It also has great support for JSON out of the box. Finally, Node.js has …
Have you ever had a situation where you found information on a webpage that you wanted to get into Splunk? I recently did and I wrote a free Splunk app called Website Input that makes it easy for everyone to extract information from web-pages and get it into a Splunk instance.
There are many cases where web-pages include data that would be useful in Splunk but there is no API to get it. In my case, I needed to diagnose some networking problems that I suspected was related to my DSL connection. My modem has lots of details about the state of the connection but only within the web interface. It supports a syslog feed but it doesn’t include …
Splunking Social Media: Tracking Tweets
So you use Twitter and have heard Splunk can do “Big Data”. By tapping into Twitter’s API you can use Splunk to investigate the stream of tweets being generated across the globe.
The great thing about using Splunk to do this is that you have complete control of the data meaning it’s incredibly flexible as to what you can build. A few basic ideas I’ve had include tracking hashtags, following specific influencers, or tracking tweets by location in real-time.
What’s more, it takes a matter of minutes before you can start analysing the wealth of data being generated. This post will show you how.…
Splunking the World Cup 2014: Real Time Match Analysis
As an Englishman I’ve been waiting months – with very high expectations – for the World Cup to come around. Reading fellow Splunker, Matt Davies’ blog post titled, “Splunking World Cup 2014. The winner will be…“, only heightened my excitement.
The tournament is now going into the second week and I’ve been starting to look at the teams, players, and tournament more closely. Which stadium holds the most people? Who’s the top scorer? Which referee hands out the most cards?
With these questions fresh in my mind I opened up Splunk and began to have a look at the huge amounts of information being streamed from the tournament. For this post I’m going to explore real-time match updates; including teams, …
The Splunk SDK for Python gets modular input support
Support for modular inputs in Splunk 5.0 and later enables you to add new types of inputs to Splunk that are treated as native Splunk inputs.
The latest release of the Splunk SDK for Python brings modular input support. The Splunk SDKs for C# (see Developing Modular Inputs in C#) and Java also have this functionality as of version 220.127.116.11 and 1.2, respectively. The Splunk SDK for Python enables you to use Python to create new modular inputs for Splunk.
The Splunk SDK for Python comes …
Making SNMP Simpler
From Wikipedia :
Simple Network Management Protocol (SNMP) is an “Internet-standard protocol for managing devices on IP networks”. Devices that typically support SNMP include routers, switches, servers, workstations, printers, modem racks, and more.
SNMP exposes management data in the form of variables on the managed systems.
The variables accessible via SNMP are organized in hierarchies. These hierarchies, and other metadata (such as type and description of the variable), are described by Management Information Bases (MIBs).
MIBs describe the structure of the management data of a device subsystem; they use a hierarchical namespace containing object identifiers (OID). Each OID identifies a variable that can be read or set via SNMP. MIBs use the notation defined by ASN.1.
SNMP agents can …
Command Modular Input
Simplifying the status quo
I’m often thinking about potential sources of data for Splunk and how to facilitate getting this data into Splunk in the simplest manner possible.
And what better source of data than existing programs on your operating system that already do the heavy lifting for you.
Now this is nothing new to Splunk , we’ve always been able to wrap up a program in a scripted input, execute it, transform the output and pipe it into Splunk.
But rather than going and creating many of these specific program wrappers for Splunk each time you need to capture a program’s output , why not create 1 single Modular Input that can be used as a generic wrapper for …
Getting data from your REST APIs into Splunk
More and more products,services and platforms these days are exposing their data and functionality via RESTful APIs.
REST really has emerged over previous architectural approaches as the defacto standard for building and exposing web APIs to enable third partys to hook into your data and functionality. It is simple , lightweight , platform independent,language interoperable and re-uses HTTP constructs. All good gravy. And of course , Splunk has it’s own REST API also.
The Data Potential
I see a world of data out there available via REST that can be brought into Splunk, correlated and enriched against your existing data, or used for entirely new uses cases that you might conceive of once you see what is available and …