At the recent Hack In The Box Conference in Kuala Lumpur, Malaysia, I was presenting on insider crime visualization. I was also showing how you can use Splunk on the command line (and through the Web interface) in order to retrieve events and process them further with your own scripts. One of the ways to use Splunk on the command line is simply the following:

./splunk search "ipfw | fields + SourceAddress DestinationAddress" -auth admin:changeme | awk ‘{printf"%s,%s",$1,$2}’ | afterglow -t -b 2 | neato -Tgif -o test.gif
What this command does, is running a search in Splunk. The search extracts only two fields from the logs, the SourceAddress and the DestinationAddress. Then you format Splunk’s output as CSV (I think there is also a command in Splunk to do so. I’d have to check) and you pipe the output into AfterGlow to visualize it.

This way of using Splunk on the command line is really powerful and opens a lot of very interesting use-cases.

In one of my old blog posts I talked about how to do a lookup of IP addresses to map them to a geo location. That time, I was showing how it is done on the command line and totally outside of Splunk. However, what I really wanted is a way to lookup the locations within Splunk whenever an IP address is shown in an event.

A lookup should take the IP address, figure out its coordinates and then plot the result on a map. What better to use than Google Earth and Google Maps. This is what my Google add-on does. The problem that I had to overcome was the mapping of the IP address to a location. I could have used some python library and put a python script into the add-on to actually do the translation. I found an easier solution by using a Web service to do the lookup for me. In addition to just translating the IP address to a geo location, the service even generates KML for me, which is the file format to describe for Google Earth where the IP address is located. The only thing I had to do is building a simple field action associated with IP addresses. After the add-on is installed, you click on the field action of an IP address. A new menu entry is then shown, which  calls an external Web service that returns the KML file. This file you then open up in Google Earth. For your reference, here is the field action from the bundle:

[googleearth-1]
metaKeys=ip
uri=http://www.someservice.com/index.php/iptokml?ip={$ip}&comment={$ip}
label=Locate in Google Earth

Simple, isn’t it?

Have you ever looked into visualizing your IT data? If you have been using Splunk 3.0 and you have been using the enhanced reporting, your answer would be yes. Maybe you have heard of, or even worked with a tool called AfterGlow. This is a tool that I co-authored with one of my old co-workers. AfterGlow is a tool that helps you visualize your IT data as link graphs. It is extremely simple to use. It is completely command line driven, uses CSV as input and generates a DOT description for a graph, according to some predefined conditions that are provided in the form of a properties file. the DOT file can then be visualized with one of many tools, such as GraphViz. You can find some examples on SecViz - look for the link graphs.

I have recently built an integration between Splunk and AfterGlow. If you are a complete command line junky, here is how to use Splunk with AfterGlow through Splunk’s CLI:

./splunk search "ipfw | fields + SourceAddress DestinationAddress DestinationPort" -auth admin:changeme | awk ‘{printf"%s,%s,%s\n",$1,$2,$3}’ | afterglow -c properities -b 2 | neato -Tgif -o /tmp/test.gif

Too complicated? Well, maybe. But keep in mind, this is extremely powerful, at each stage in the pipeline, you can change the data to whatever you need it. You also don’t need anything else than a working Splunk and AfterGlow installation. If you would prefer a nicer solution, I built a command for Splunk to hide the afterglow logic. What you have to do is the following:

1. Download AfterGlow. Make sure you are using Version 1.5.9 (It’s available, I just didn’t officially announce it yet ;(
2. Install Splunk and install my AfterGlow Bundle.
3. If you downloaded AfterGlow 1.5.9, you don’t need to patch AfterGlow, otherwise do so as described in the README.
4. Make sure you have GraphViz installed and it is working.
5. UPDATE: Go into $SPLUNK_HOME/etc/searchscripts and edit afterglow.py:
   • Update the afterglowLocation to where you installed afterglow
   • Update the afterglowParameter, use AfterGlow 1.5.9 and update the path to your configuration file
   • Update the targetDir location to point to your Splunk install’s oxiclean/static directory.
   • Make sure your afterglow properties file contains the following line:

   url=http://localhost:8000/?q=\N%20starthoursago%3A%3A24

   • I hope this will fix things!
6. Construct a search in Splunk that returns three fields. For example
   ipfw | fields + SourceAddress DestinationAddress DestinationPort

I did this by using the ipfw bundle. The idea is simply that you use the fields command to extract two or three fields from your log entries. Once you have this working, simply add: “| afterglow” to your search. This will pipe the search result through the new afterglow command. You should now get one event back which is comprised of just a single field. Click on the associated action and choose: “Show AfterGlow Graph”. This should open a new window which contains a link graph with your data. Also note how you can click on the nodes and it will take you back in to Splunk and search for that node’s value. This completes a complete bi-directional integration between the two tools.

Let me know how this works for you and share your graphs with me! Even better, put your graphs on SecViz!

UPDATE: If you are getting errors, check the comments for this post!

Hello,

I am one of the new faces running around in the Splunk offices. The purpose of this blog post is to introduce myself. What am I doing at Splunk and what have I been doing in my past? I joined about a month ago to work in product management. My area of responsibility is the user interface, as well as search and indexing, which includes things like the search language. I am also taking on some of the responsibilities for Splunk’s “solutions” in the security area.

I most recently worked at ArcSight, where I had the responsibility over all of the solutions spanning three product lines. The three product lines were the enterprise security manager, which is a high-end correlation engine, the logger appliance - basically a very fast log collection appliance, the networking products dealing with network configuration management, as well as the threat response manager. The solutions addressed anything from regulatory compliance (SOX, PCI, IT governance, HIPAA, FISMA) to insider threat management and hardening guidelines for network equipment.

Prior to ArcSight, I worked for IBM research in Switzerland. I did a lot of work in the area of intrusion detection, correlation, and testing of IDSs. I also used to work for PriceWaterhouse Coppers Consulting, where I was an IT security consultant.

Approximately three years ago, ArcSight built a feature into the SIM product that allowed the user to visualize security data. I was magically drawn to it and started playing around. It was around the same time that the first user conference was going to be held. More out of stupidity than anything else, I submitted a proposal for a visualization talk. It got accepted and I had to put something together on the topic of visualizing security data. The presentation was a huge success. The audience loved the new way of dealing with their security data. They saw an incredible efficiency boost. That was the birth of my passion for security data visualization. Sine then I have presented at conferences around the world on the topic.

Slightly after the user conference, together with a co-worker, Christian, I started an open source project called AfterGlow. AfterGlow is a tool that helps you visualize data as link graphs. The obsession with visualization got as far as me currently writing a book about the topic. It will be published by Addison Wesley, probably called “Applied Security Visualization”. The story how I got started writing a book is another long one and I will keep that for another blog post

Quite a few people have asked me why I left ArcSight. The reasons are a few. Looking from a pure product standpoint, there are some very interesting differences between IT data search and the traditional SIM way of managing log data. I am going to address the topic in some future blog posts. On my personal blog I already started to outline the problem of normalization, which is probably the biggest and most important diffference. I will roll the topic up again right here and continue some of the discussions around the differences between Splunk and the rest of the log management space.

I am extremely excited to be part of the Splunk team.