Integrating with Splunk: You Gotta Think Outside the Box

This morning, a question was asked about integrating with Splunk that started with something like, “but I can’t send syslog from my system, so how can I get that data in Splunk?” It really doesn’t matter what system or what data; before digging in, I already knew that the answer was out there.

“But wait a second, Hal, how could you know that?”, you might be thinking.

Well, it’s just a matter of knowing a bit about how computer systems work, and understanding that Splunk has many ways of ingesting data. You see, at a very high level, there are only two ways that Splunk can integrate with another system. I’ll call these integration types “intentional”, and “operational”. Let’s define them:

  1. Intentional – this is when a system causes machine data to be emitted intentionally, for the purpose of exchanging that data with another system. Examples might include:
    1. sending a syslog event
    2. sending an SNMP trap
    3. sending an email
    4. triggering any action, such as invoking a script
    5. APIs, SDKs, scripting toolkits
  2. Operational – this is when machine data is created, but not necessarily emitted, simply due to the normal operations of a system. This one is key to understand, because these may be the important sources of data that you didn’t know you even had. Examples:
    1. log files
    2. databases
    3. message queues or busses

Nine times out of ten*, people start with syslog events and end with log files. But there is so much more out there! Let’s say that you have some piece of software which can send emails when something important happens. You like that software, it’s not going anywhere, but you really need to get that important event into Splunk. You’ve already looked at syslog and log files, and don’t know where to go next? Not a problem, we got you! Here are some ideas that might help out:

I don’t mean to bury the lead, but I wanted this post to be general in nature. For those curious, the system in question was Cisco Prime Infrastructure. I’ve never used it, but was able to determine that the above techniques had a good chance of working after skimming the admin guide.

Happy Splunking!

(* I totally made up this statistic.)

On cisco prime infra another “Operational ” type is the prime REST API that can be accessed through the rest modular input app.

June 16, 2015

Thanks Simon, great tip! I looked more into this and found that it’s quite comprehensive in fact and would be the optimal route to success. Here are the Cisco Prime Infrastructure API docs for the next person to come across this post:

And BTW, I would actually put an API in the intentional bucket for sure. APIs and SDKs are certainly made on purpose for integration and interopability.

June 17, 2015

Can we send real time squid access.log data to splunk?
I have already installed forwarder on the squid machine. But am lost as to what the next step is to get it to monitor /var/log/access.log and send new data over to splunk ?

January 3, 2016

I suggest reading some of the Getting Data In chapter in our docs. You may also want to check out this Squid App for Splunk on Splunkbase, it looks interesting. If you get stuck, ask a specific question here:

January 4, 2016

One Trackback

  1. […] Integrating with Splunk: You Gotta Think Outside the Box (Hal Rottenberg) […]