Real time Machine Diagrams in Splunk, Part I

I’ve always wanted a way to integrate Splunk machine data with actual diagrams of our network. The idea is to reproduce the look and feel of a computerized system control panel. In the control panel, icons represent components of the system, and as the state these components change, aspects of the diagram change to reflect their significance (icons turn green, red, etc):

nuclearPowerPlantSimPanel

Here’s a screen capture of my attempt to create something like this for Splunk.

I call it a ‘Real time Machine Diagram’:

Real time Machine Diagram

Using a Real-time search, this diagram dynamically shades icons representing machines on network. As the error rate of a machine increases (the number of errors coming across in logs increases) the icons turn intensely red; as the error rate …

» Continue reading

How to ease your System Administration email workload with Splunk

Systems generate a lot of email. If you’re a sysadmin, you already know this. If you work with sysadmins, then you’re to blame (ok, maybe not). In either case, dealing with system email is time consuming, and the signal-to-noise ratio is low. More often than not these emails are ignored (procmail FTW!).
Is this a good thing? No.
Why?
These emails are generated for a reason, and that reason is usually that there’s something amiss on your system. Instead of /dev/null’ing all of these useful nuggets, why not mine them with Splunk?

In this How To we’ll setup a catch-all Postfix server and use it to Splunk all of your system generated email.…

» Continue reading

How to use Notifo to receive Splunk alerts on your iPhone

In this article I’ll describe how I use Splunk and Notifo to alert me whenever someone tries to login to my system with invalid credentials. Notifo is push-based notification service for mobile phones, in our example we’ll be using the iPhone.

Overview

  1. Setup a Notifo account.
  2. Install the Notifo app on your iPhone.
  3. Install the notifo.py Python module.
  4. Install the splunknotifo.py Python alert script.
  5. Setup splunknotifo.py
  6. Setup saved search.

Assumptions

  • This process assumes that you’ve got Splunk installed and monitoring a file containing sshd log messages.

Steps…

» Continue reading

Howto post Splunk saved search results to Twitter

Overview

  • Create a saved search in Splunk.
  • Upload the tweet.sh script to your Splunk server.
  • Schedule your saved search in Splunk Manager.

Steps

  1. Copy the script below to your Splunk server under $SPLUNK_HOME/bin/scripts/, replace TWITTERUSER and TWITTERPASS with your Twitter username and password respectively.
    #!/bin/sh
    # tweet.sh - Post Splunk saved search result to Twitter.
    # Greg Albrecht (gba@splunk.com)
    # (c)2010 Splunk, Inc.
    if [ -f "$9" ]; then
    export IFS=$'\n'
    for MSG in $(gzcat $9 | cut -d , -f 5|grep -v "_raw"|sed s/^\"//g|sed s/\"$//g); do \
    /usr/local/bin/curl --basic --user "TWITTERUSER:TWITTERPASS" --data-ascii "status=$MSG" http://twitter.com/statuses/update.json \
    ;done
    fi
  2. Login to your Splunk instance and navigate to the search app
  3. Create a search for the terms events you want to
» Continue reading

How to consume tcptrace with Splunk 4.0

The idea to consume tcptrace with Splunk came to me after seeing Darren Hoch‘s OSCON 2009 presentation Linux System and Network Performance Monitoring. In his talk Darren shows how he diagnosed home networking issues using tcptrace. Here’s his description of tcptrace:

The tcptrace utility provides detailed TCP based information about specific
connections. The utility uses libpcap based files to perform an analysis of
specific TCP sessions. The utility provides information that is sometimes difficult
to catch in a TCP stream. This information includes:
• TCP Retransmissions – the amount of packets that needed to
be sent again and the total data size
• TCP Window Sizes – identify slow connections with small
window sizes
• Total throughput

» Continue reading

Eating NetFlow with Splunk, Part 1

It’s easy to eat network data using Splunk. In a recent seminar I demonstrated how quickly a network administrator could dig through NetFlow data to diagnose network problems using Splunk. Here I’ll show you some steps for getting NetFlow (cflow, jflow, netstream, IPFIX, sflow) data into Splunk.…

» Continue reading