Yesterday marked yet another milestone in my life as an author. I got the first 5 chapters of my book back from production. The Applied Security Visualization book is slowly coming together. After working on the book for one and a half years, it is great to finally see how the book is going to look. The graphs are placed on the pages and the layout is done. It finally feels like a real book. The book will be out by BlackHat at the beginning of August.

.

.

You can pre-order the book on Amazon. It is about 400 pages and contains the following chapters:

1. 1. Visualization

2. 2. Data Sources

3. 3. Visually Representing Data

4. 4. From Data to Graphs

5. 5. Visual Security Analysis

6. 6. Perimeter Threat

7. 7. Compliance

8. 8. Insider Threat

9. 9. Data Visualization Tools

The book ships with a live visualization CD. DAVIX, the data analysis and visualization UNIX, contains all the visualization tools discussed in chapter 9. They are all readily installed so you can use them to visualize your own data. No need to go through any crazy installation processes. The Web site for DAVIX is going to be ready by BlackHat, where we will officially launch DAVIX. If you are interested in a pre-version, drop me an email.

I was giving a talk at SOURCEBoston 2008. The topic this time was around general visualization and what has gone wrong in security visualization in the past. I showed how we can learn and steal from other disciplines, in this case, the New York Times. The NYT has done some pretty fantastic work in the area of data visualization. Their interactive market map, for example, is a great way of exploring stock data. During the talk, I outlined some of the design principles that the NYT graphics department is using when they are designing their graphs: Show - Don’t Tell.

To start my presentation, I showed a little video about security visualization (see below).

At conferences lately, I find myself not to be the only one that talks about security visualization. More and more presentations are showing visualizations. A lot of projects are using visualization to help them analyze all the data at hand. At SOURCE, Dave Dittrich from the University of Washington, talked about BotNet analysis and visualizing network traffic captured from BotNets. He definitely has a challenge of displaying large amounts of data. We discussed some approaches and possibly, parallel coordinates, could work for his data. Parallel coordinates are what I used in my book for some BotNet traffic analysis.

For the past year I have been working on a book about visualization. It will be called “Applied Security Visualization“. The book is going to talk about all the aspects of visualizing security data. Anything from important data sources and graphs to use-cases and open source tools for visualization. The main use-cases I write about evolve around Perimeter Threat, Compliance, and Insider Threat.

Last year during RSA, Addison-Wesley (my publisher) recorded some videos, where I talk about the book and some of its contents. Here are the links to the videocasts:

• Security visualization
• Bridging security and visualization

At this point, I have one more chapter to write before the book is done. A rough-cut version should be available by RSA this year and the book should be out by BlackHat (August). Keep your fingers crossed!

At the recent Hack In The Box Conference in Kuala Lumpur, Malaysia, I was presenting on insider crime visualization. I was also showing how you can use Splunk on the command line (and through the Web interface) in order to retrieve events and process them further with your own scripts. One of the ways to use Splunk on the command line is simply the following:

./splunk search "ipfw | fields + SourceAddress DestinationAddress" -auth admin:changeme | awk ‘{printf"%s,%s",$1,$2}’ | afterglow -t -b 2 | neato -Tgif -o test.gif
What this command does, is running a search in Splunk. The search extracts only two fields from the logs, the SourceAddress and the DestinationAddress. Then you format Splunk’s output as CSV (I think there is also a command in Splunk to do so. I’d have to check) and you pipe the output into AfterGlow to visualize it.

This way of using Splunk on the command line is really powerful and opens a lot of very interesting use-cases.

In one of my old blog posts I talked about how to do a lookup of IP addresses to map them to a geo location. That time, I was showing how it is done on the command line and totally outside of Splunk. However, what I really wanted is a way to lookup the locations within Splunk whenever an IP address is shown in an event.

A lookup should take the IP address, figure out its coordinates and then plot the result on a map. What better to use than Google Earth and Google Maps. This is what my Google add-on does. The problem that I had to overcome was the mapping of the IP address to a location. I could have used some python library and put a python script into the add-on to actually do the translation. I found an easier solution by using a Web service to do the lookup for me. In addition to just translating the IP address to a geo location, the service even generates KML for me, which is the file format to describe for Google Earth where the IP address is located. The only thing I had to do is building a simple field action associated with IP addresses. After the add-on is installed, you click on the field action of an IP address. A new menu entry is then shown, which  calls an external Web service that returns the KML file. This file you then open up in Google Earth. For your reference, here is the field action from the bundle:

[googleearth-1]
metaKeys=ip
uri=http://www.someservice.com/index.php/iptokml?ip={$ip}&comment={$ip}
label=Locate in Google Earth

Simple, isn’t it?

Have you ever looked into visualizing your IT data? If you have been using Splunk 3.0 and you have been using the enhanced reporting, your answer would be yes. Maybe you have heard of, or even worked with a tool called AfterGlow. This is a tool that I co-authored with one of my old co-workers. AfterGlow is a tool that helps you visualize your IT data as link graphs. It is extremely simple to use. It is completely command line driven, uses CSV as input and generates a DOT description for a graph, according to some predefined conditions that are provided in the form of a properties file. the DOT file can then be visualized with one of many tools, such as GraphViz. You can find some examples on SecViz - look for the link graphs.

I have recently built an integration between Splunk and AfterGlow. If you are a complete command line junky, here is how to use Splunk with AfterGlow through Splunk’s CLI:

./splunk search "ipfw | fields + SourceAddress DestinationAddress DestinationPort" -auth admin:changeme | awk ‘{printf"%s,%s,%s\n",$1,$2,$3}’ | afterglow -c properities -b 2 | neato -Tgif -o /tmp/test.gif

Too complicated? Well, maybe. But keep in mind, this is extremely powerful, at each stage in the pipeline, you can change the data to whatever you need it. You also don’t need anything else than a working Splunk and AfterGlow installation. If you would prefer a nicer solution, I built a command for Splunk to hide the afterglow logic. What you have to do is the following:

1. Download AfterGlow. Make sure you are using Version 1.5.9 (It’s available, I just didn’t officially announce it yet ;(
2. Install Splunk and install my AfterGlow Bundle.
3. If you downloaded AfterGlow 1.5.9, you don’t need to patch AfterGlow, otherwise do so as described in the README.
4. Make sure you have GraphViz installed and it is working.
5. UPDATE: Go into $SPLUNK_HOME/etc/searchscripts and edit afterglow.py:
   • Update the afterglowLocation to where you installed afterglow
   • Update the afterglowParameter, use AfterGlow 1.5.9 and update the path to your configuration file
   • Update the targetDir location to point to your Splunk install’s oxiclean/static directory.
   • Make sure your afterglow properties file contains the following line:

   url=http://localhost:8000/?q=\N%20starthoursago%3A%3A24

   • I hope this will fix things!
6. Construct a search in Splunk that returns three fields. For example
   ipfw | fields + SourceAddress DestinationAddress DestinationPort

I did this by using the ipfw bundle. The idea is simply that you use the fields command to extract two or three fields from your log entries. Once you have this working, simply add: “| afterglow” to your search. This will pipe the search result through the new afterglow command. You should now get one event back which is comprised of just a single field. Click on the associated action and choose: “Show AfterGlow Graph”. This should open a new window which contains a link graph with your data. Also note how you can click on the nodes and it will take you back in to Splunk and search for that node’s value. This completes a complete bi-directional integration between the two tools.

Let me know how this works for you and share your graphs with me! Even better, put your graphs on SecViz!

UPDATE: If you are getting errors, check the comments for this post!