AfterGlow and Splunk
| Topics: | Homepage, Log Analysis, Visualization |
|---|---|
| Tags: | |
| Share: |
Have you ever looked into visualizing your IT data? If you have been using Splunk 3.0 and you have been using the enhanced reporting, your answer would be yes. Maybe you have heard of, or even worked with a tool called AfterGlow. This is a tool that I co-authored with one of my old co-workers. AfterGlow is a tool that helps you visualize your IT data as link graphs. It is extremely simple to use. It is completely command line driven, uses CSV as input and generates a DOT description for a graph, according to some predefined conditions that are provided in the form of a properties file. the DOT file can then be visualized with one of many tools, such as GraphViz. You can find some examples on SecViz - look for the link graphs.
I have recently built an integration between Splunk and AfterGlow. If you are a complete command line junky, here is how to use Splunk with AfterGlow through Splunk’s CLI:
./splunk search "ipfw | fields + SourceAddress DestinationAddress DestinationPort" -auth admin:changeme | awk ‘{printf"%s,%s,%s\n",$1,$2,$3}’ | afterglow -c properities -b 2 | neato -Tgif -o /tmp/test.gif
Too complicated? Well, maybe. But keep in mind, this is extremely powerful, at each stage in the pipeline, you can change the data to whatever you need it. You also don’t need anything else than a working Splunk and AfterGlow installation. If you would prefer a nicer solution, I built a command for Splunk to hide the afterglow logic. What you have to do is the following:
- Download AfterGlow. Make sure you are using Version 1.5.9 (It’s available, I just didn’t officially announce it yet ;(
- Install Splunk and install my AfterGlow Bundle.
- If you downloaded AfterGlow 1.5.9, you don’t need to patch AfterGlow, otherwise do so as described in the README.
- Make sure you have GraphViz installed and it is working.
- UPDATE: Go into $SPLUNK_HOME/etc/searchscripts and edit afterglow.py:
- Update the afterglowLocation to where you installed afterglow
- Update the afterglowParameter, use AfterGlow 1.5.9 and update the path to your configuration file
- Update the targetDir location to point to your Splunk install’s oxiclean/static directory.
- Make sure your afterglow properties file contains the following line:
- I hope this will fix things!
url=http://localhost:8000/?q=\N%20starthoursago%3A%3A24 - Construct a search in Splunk that returns three fields. For example
ipfw | fields + SourceAddress DestinationAddress DestinationPort
I did this by using the ipfw bundle. The idea is simply that you use the fields command to extract two or three fields from your log entries. Once you have this working, simply add: “| afterglow” to your search. This will pipe the search result through the new afterglow command. You should now get one event back which is comprised of just a single field. Click on the associated action and choose: “Show AfterGlow Graph”. This should open a new window which contains a link graph with your data. Also note how you can click on the nodes and it will take you back in to Splunk and search for that node’s value. This completes a complete bi-directional integration between the two tools.
Let me know how this works for you and share your graphs with me! Even better, put your graphs on SecViz!
UPDATE: If you are getting errors, check the comments for this post!
September 16th, 2007 at 11:39 pm
[...] As you might have seen on secviz.org, AfterGlow 1.5.9 is out. The announcement of AfterGlow 1.5.9 on secviz has some more details on what’s new. Just quickly here: The URL feature is pretty interesting and addresses some old thoughts and things I have been talking about with other people (Peter, are you reading this?). The issue there was that the AfterGlow graphs are very static and that’s kind of a bummer. It would be really nice if there was more interactivity. Clicking on nodes for example. Well, this is now a first step towards that. Along with the Splunk - AfterGlow integration, this is actually going almost all the way of completing the interaction round trip. I know, in terms of real interactivity, there is still a lot missing, but I think this is taking care of some really interesting use-cases. [...]
September 17th, 2007 at 4:25 pm
If you are getting an error when running the search command in Splunk, try running afterglow on the command line:
cat file.csv | afterglow.pl | neato -Tgif -o test.gifCan't locate Text/CSV.pm in @INC ...it means that you need to install the Perl Text::CVS module. In Ubuntu you use:
aptitude install libtext-csv-perl
-bash: neato: command not found
you will have to install GraphViz from http://www.graphviz.org
September 27th, 2007 at 1:28 pm
Raffy,
Looks really cool. Thanks for putting this together. I look forward to learning more.
-D
October 4th, 2007 at 9:40 am
Raffy,
I followed the directions above but could not get past the part that says to edit the /opt/splunk/etc/serachscripts/afterglow.py because it does not exist. Is it included in the 1.5.9 distro or do I have to create it for this to work?
T
October 4th, 2007 at 6:17 pm
tmessner, the file will only exist if you unpacked the afterglow bundle. It’s not part of the afterglow distro.
The bundle is on splunk base: http://www.splunkbase.com/addons/All/Splunk/addon:AfterGlow_Graphing
Hope this works.
October 18th, 2007 at 10:25 am
>install my AfterGlow Bundle.
How do you actually install the add-on? I can’t find it anywhere…
October 22nd, 2007 at 9:03 am
Sorry, I guess there’s a naming problem. I called the add-on a bundle. SORRY! Here is the link again for the add-on: http://www.splunkbase.com/addons/All/Splunk/addon:AfterGlow_Graphing
January 9th, 2008 at 10:08 pm
I’m sure you’ve been told this before, but this is a phenomenal plugin; I just got it working and I’m forcing myself to stop fiddling with it and get some sleep. One thing I noticed that might impact other users:
1. Splunk fails to process the output under some circumstances (using RHEL5 64-bit and the latest Splunk preview here)because the afterglow.py command outputs some extraneous messages to the screen, like this:
event :: eventLabelExp
getLabel:
target :: targetLabelExp
Splunk interprets these as search results, so the search doesn’t complete properly.
The ugly, but quick hack around this I used was to modify the beginning of this line:
output = os.popen(command + ” | ” …
to this:
output = os.popen(command + ” 2>/dev/null | ” …
thus, simply dumping the output. There may be a simpler way to do this via a command-line switch for afterglow.py itself, but I’ll confess to not having read all the documentation yet.
Great work!
June 12th, 2008 at 12:37 am
Hi,
nice work, Raffy. I had the same issue as James and its workaround fixed it and now I can see the graph from within splunk web. I’m using ubuntu hardy in i386.
By the way, the generated graph doesn’t seem to have an imagemap in the HTML file. There is a .map file, though.
Thanks for the bundle!
June 27th, 2008 at 11:16 am
After installing I realize that we must have missed something because when we add | afterglow to our search we see this error:
An error occurred during search: Error in ‘SearchParser’: Unknown search operation: ‘afterglow’
We’re running Ubuntu Server and Splunk is installed in /opt/splunk so we copied the afterglow bundle folder (extracted) to /opt/splunk/etc/bundles/afterglow.
We edited the afterglow.py script and entered the correct values.
Any help would be great!
Thanks,
HSPCD
June 27th, 2008 at 11:34 am
Are you, by any chance running 3.3 preview? You have to edit the commands.conf file and the authorize.conf file as well. In both files you need to add the afterglow script.
June 27th, 2008 at 11:38 am
I am using 3.2.3 version.
June 27th, 2008 at 11:40 am
Oh, in 3.2.x you have to copy the afterglow.py into your etc/searchscripts directory. I am sure that’s the problem you are seeing.
August 13th, 2008 at 3:32 pm
I am runnning on Ubuntu Server. Everything works correctly, however, after generating a graph, the results returned to splunk consist of lots of entries like source :: sourceLabelExp=getLabel:, source :: sourceLabelExp=target :: targetLabelExp, etc. Finally at the bottom of the results, I see afterglowFilename=afterglow.html, which i can select to view the graph. Am I missing something here? Any help with the issue is appreciated. Thanks!
August 13th, 2008 at 4:40 pm
jdb, what you are seeing is an artifact of me posting a version of AfterGlow that has debugging in there. I have a version 1.6 ready that takes it out. Look at Jame’s suggestion further up in the comments to get rid of that too. I will post 1.6 at some point. You can get it from CVS if you want. Sorry for the inconvenience.