Browsing over on Wikipedia, one excerpt states that “a platform describes some sort of hardware architecture or software framework”, and the description for a software framework, says it “may include support programs, code libraries, a scripting language, or other software to help develop and glue together the different components of a software project”.

A platform can be considered as a type of framework - one which helps developers write software faster by a) giving them the tools to develop against it, and b) transparently dealing with the under-the-hood, nitty-gritty work necessary when dealing with difficult problems. Difficult problems like indexing and searching gigabytes upon gigabytes of event data, for example.

Well, that’s exactly what the Splunk Platform does for developers. It provides resources, examples, and SDKs for developing a variety of applications around the robust Splunk engine, and it provides a launching point for domain specific development, from availability and security, to business intelligence and compliance.

BTW, this isn’t something we’re just waving our hands around about and saying “look at this white paper, isn’t this a nice idea?”. Nope. Platform is here, and it’s here today, with links to real code, real content, and real resources for the developers looking to write the next great idea.

Here’s how to get started writing your first application against the Splunk Platform:

1. 1. Download a Preview of Splunk that has a brand new shiny REST-based API built into it.
2. 2. Head on over to the developer’s wiki and start digging around in the API howtos.
3. 4. Download the new .NET SDK from its Google Code project page.
4. 4. Join the any of the projects and start contributing code/content.
5. 5. Join the new Splunk Labs list and start interacting (asynchronously) with our developers.
6. 6. Hop on #splunk on IRC and chat with us in real time.

We’ll be continuing to add content and resources to the Platform effort, and we encourage your participation in the development community as it forms.

Two things struck me interesting about Charlie’s post.

First, he noticed the changes in the UI we’ve been slowly making over the last few releases. If you’ve ever done UI design, you know how much sweat goes into every little detail, and how much momentum a design carries over time. That someone noticed the new changes *and* liked them is a HUGE win for the UI team. It’s even better how fast someone noticed!

Second, he actually spends quite a bit of time explaining the security workaround in the free product - one that I covered earlier, coincidently enough. I figure if someone goes to the time and trouble to figure out how they can keep using the product in a secure, legitimate way, then they must really, really like it. You simply can’t argue with an evangelist like this.

If anyone here is a gem, it’s Charlie.

What you really want to do is refer to them with different subdomain names, where something like http://splunkpreview.mydomain.com/ would bring up Splunk without having to remember the port number.

If you are running Apache, (like I am on Leopard) you get a reverse proxy server for free. With just a few lines of configuration, you can alias subdomains (or domains for that matter) to your heart’s content.

You also get the ability of putting content behind some basic authentication provided via Apache’s HTTP auth methods. This comes in handy if you’d like to link to your Splunk install from a publicly facing page, but don’t want people to know what type of content is behind the authentication. It also works for limiting access to a particular IP address group or domain.

I’ve put together a screencast covering how to do this from OS X’s version of Apache. Click on the thumbnail below to play the screencast.

Note: Firewalling the actual port Splunk runs on is left as an exercise for the viewer, as is limiting access to a group of IP addresses. More information about configuring Apache’s mod_proxy module can be found on Apache’s website.

Here’s the configuration code from the screencast:

I mentioned this in passing to one Sean Dick who is a developer friend of mine in Oklahoma City. What follows is a nearly identical post to the one he made over at his self-named blogpost on Blogger on how to get Rails to integrate with Splunk. “There’s plenty left to do.”, he said, but I’m convinced it’s worthy of mentioning here. Thanks for hammering this out Sean!

Serious Material from Sean Begins Here

As per the norm, this post assumes you’ve downloaded Splunk for your particular platform. It also requires a newer install of Ruby on Rails. Come back when you’ve completed both these tasks.

Get Splunk started now:

> sudo export SPLUNK_HOME=/opt/splunk/
> sudo ./opt/splunk/bin/splunk start


You need to drop the Splunk on Rails plugin into your Rails app’s lib folder and then call it with require ’splunkbase’ Note: If you’re running Splunk remote or on a non-standard port, don’t forget to change the SERVER variable in the plugin file!

Now let’s say you want to make sure your rails application is running bug-free, and when one does pop up, you need to know it pronto. You’ll create a new controller into which we’re going to put some splunk goodies. I named mine SplunkController, but you can be more creative.


class SplunkController < ApplicationController
require 'splunkbase'
@@foo = SplunkBase.new
def index
end
def reports
@document =" @@foo.splunkSearch('q' => params[:query])
end
end


This is really nothing more than making available the response from splunk to your view in the form of a variable. Defining a page for it to be displayed in is no more difficult than:

<pre><%= @document %> </pre>


Now we build the index page we defined so we can pass it the query:

<html>
<head>
<%= javascript_include_tag "prototype" %>
</head>
<body >
<%= form_remote_tag(:update => "graphDiv",
:url => {:action => :reports }) %>
<%= text_field_tag :query, nil, {:size => "100"} %>

<%= submit_tag "Get a report on your query" %>
<%= end_form_tag %>
<div id="graphDiv">
</div>
</body>
</html>


And believe it or not we’re ready to start asking Splunk some questions. Try giving it something like:

[search sourcetype::what_you_named_your_source error starthoursago=24] | outputxml


As you’ve probably gathered, that’ll give you a formatted list of all of the errors that have occurred in the last day.

Now let’s say you’ve got a rails application running internally that you don’t have the option to/don’t feel comfortable with outsource analysis to something like Google Analytics. Back to our cute little controller, we add in a new definition for the graphing page.

class SplunkController < ApplicationController
require 'splunkbase' #those two magical words
@@foo = SplunkBase.new
def index
end
def reports
@document = @@foo.splunkSearch('q' => params[:query])
end
def graph #for graphing, this fixes things up so we can display the data
@datahash = {} @queryDoc= @@foo.splunkSearch(’q’ => params[:query]) #here is the meat
@queryDoc.each_element(”//r/”) do |ele| #here we’re sorting out what is useful
@datahash[ele.elements["m[@col='1']“].text] = ele.elements["m[@col='2']“].text.to_i
end
@sorted = @datahash.values.sort.reverse #sorting it for the hell of it
@chartheight = @datahash.values.max + 50 #to make it look pretty and consistent
end
end


This example is fairly simple and assumes you’re just looking for basic metrics on your site’s usage. You could build it larger to accept whatever you want splunk to throw back at you. This one expects to see something like "Content Name" => "value".

Now let’s take a stab at setting up the graph:


<samp><%= @queryDoc.to_s %></samp> #gives us a raw return of the data we pulled from Splunk
</div>
<% @sorted.each do |name, height| %> #Iterate through each of the data pairs and grab the height.
<div class="columnSpacer">
<div style="margin-top: <%= 100 - ((height * 100)/@chartheight.to_f) %>%"class="graphTitle">
<%= name %>
<br>
<%= height %> hits
</div>
<div class="graphColumn" style="height: <%= (height * 100)/@chartheight.to_f)%>%">

</div>
<% end %>


In the interest of keeping things from getting too esoteric I’ve committed a no-no and left some programming in the view. All in all, it’s pretty light math to get things displaying properly. As you should be able to glean from the code presented we’re just iterating through each of the name/value pairs we extracted from the XML Splunk returned and turning them into pretty little bars on a chart. Now all we need to do is put together the index page for accessing the graph function.


<html>
<head>
<%= javascript_include_tag "prototype" %>
</head>
<body >
<%= form_remote_tag(:update => "graphDiv",
:url => {:action => :graph }) %>
<%= text_field_tag :query, nil, {:size => "100"} %>

<%= submit_tag "Get a report on your query" %>
<%= end_form_tag %>
<div id="graphDiv">
</div>
</body>
</html>


There’s pretty minimal monkey business here, so let’s go on to the fun part:

Let’s take a look at what controllers are getting the most face-time for our users and what content sections are being perceived as being the most useful. This example comes from a site I did recently for a client and happens to be the most handy rails logfile I have within reach.

Pop in the query:

[search sourcetype="the_name_you_gave_your_source" | top 5 controller ] | outputxml


And you get something like this:

There are a myriad of options available to you through Splunk’s search interface, and learning to romance the queries to give you what you want would be a section all its own. This one, however consists of limiting the scope of the search ( sourcetype= ) and giving it a context to put it in ( top 5 controller ) — in this case, the top five controllers.

Next post I will cover the possibilities afforded with the use of bundles in Splunk in conjunction with your Rails application. In the meantime I highly suggest you peruse the REST API documentation supplied in your Splunk install and the admin/developer documentation on Splunk.com to get a more in-depth understanding of what you can do.

Let’s get dirty. Go into settings..general..auto-lock and set locking to ‘never’. This will keep the phone on while you hack around on it. Keeping the phone on and connected to the network will drain your battery like nobody’s business, so make sure you plug in the charging cable.

Now install AppTap. Follow the instructions, and come back here when you are all done.

Using the AppTap installer on the phone, install the Community Sources, BSD Subsystem, Term-vt100, OpenSSH, Tinyproxy, and UIctl apps, in that order. UIctl will let you stop and start sshd on the phone. Launch it now to see if sshd is running. Click on the ‘load’ button if it’s not.

Ping your phone from your computer with its IP address. You can use the terminal on the phone to grab the IP address:


# ifconfig en0
en0: flags=8863 mtu 1500
inet 10.0.1.194 netmask 0xffffff00 broadcast 10.0.1.255
ether 00:1c:b3:f0:0b:a6
#


Ssh to the phone from your terminal. The default root password is ‘dottie’.


foobar:~ kord$ ssh root@10.0.1.194
root@10.0.1.194's password:
Last login: Wed Oct 10 13:45:22 2007 from 10.0.1.191
# hostname
Kord's iPhone
#


Now add a syslog.conf file to /etc/:


bash-3.2# echo "*.* @10.0.1.191" > /etc/syslog.conf
bash-3.2# cat /etc/syslog.conf
*.* @10.0.1.191


Obviously, you’ll want to use the IP address of the machine on which you are going to install Splunk. Speaking of Splunk, at this point you should already have it installed. If you don’t, download it here, and install it now. You can reference my first hack for instructions on getting Splunk up and running quickly on your system. Smile. Splunk goooood.

Back in your ssh session to the iPhone, you’ll need to move the syslogd executable to an alternate location, kill the old instance, and start the new one with a few parameters.


# cd /usr/sbin/
# mv syslogd syslogd.mine
# launchctl stop com.apple.syslogd
....wait for about 5 seconds....
# /usr/sbin/syslogd.mine -bsd_out 1 &


Syslogd should now use the new /etc/syslog.conf file that you just created when it starts up. You can check if it’s running properly:


# ps -ax |grep syslog
110 p0 S 0:02.91 /usr/sbin/syslogd.mine -bsd_out 1
#


Now fire up Splunk, and hit your instance of it in a browser: http://localhost:8000. Click on the ‘admin’ link in the top right, click on the ‘data inputs’ tab at the top, ‘network ports’ just below that, and then click on the ‘add input’ button to the right.

Click on the UDP radio button under ’source’. The port listed should change to 514. Click on the ‘add’ button at the bottom. You should now be getting data coming into Splunk on UDP port 514. Grab some coffee whilst Splunk eats ALL the logfiles coming in from the iPhone.

Now let’s get Tinyproxy serving requests for Safari on the phone and logging through syslogd. Check that Tinyproxy is running on the iPhone first:


# ps -ax |grep tiny
354 ?? S 0:00.10 /usr/bin/tinyproxy
355 ?? S 0:00.00 /usr/bin/tinyproxy
1428 p1 S+ 0:00.01 grep tiny


Edit tiny’s configuration file to set his logs to go to syslogd. Keep in mind there is more to the config file than the few lines that I’m showing.


# vi /usr/local/etc/tinyproxy/tinyproxy.conf
~
# log only errors
#Logfile "/var/log/tinyproxy.log"
#LogLevel Info
Syslog On


Now on the iPhone, go to settings..wifi networks....http proxy. Enter the host as 127.0.0.1 and the port as 8080, just as you see in the screenshot below:

Lastly, kill Tinyproxy so he’ll start logging correctly. He restarts automagically, so all you need to do is kill the process ids:


# ps -ax |grep tiny
354 ?? S 0:00.11 /usr/bin/tinyproxy
355 ?? S 0:00.05 /usr/bin/tinyproxy
1651 p1 S+ 0:00.01 grep tiny
# kill -9 354 355
# ps -ax |grep tiny
1654 ?? S 0:00.01 /usr/bin/tinyproxy
1655 ?? S 0:00.00 /usr/bin/tinyproxy
1657 p1 S+ 0:00.02 grep tiny
#


That should be about it. You should have Splunk filling up with logs that contain web requests being requested by the Safari browser on your iPhone. Don’t forget to restore the syslog plist file, reboot, and fix it to lock after a few minutes timeout.

A friend of mine, Sean Dick, showed me a version of this idea using Splunk on Linux and a program called ‘apci’. As I’m a Mac fanboy of sorts, I dug up a shell script for the Mac that will print out a single logfile-like line containing laptop battery information, including amp draw, amp-hours left, and more. It’s aptly named ‘battery’, and you can download it here.

I suggest you put battery in a directory under your home directory, say something called ’scripts’. Head into ‘terminal’ to start the dirty work.

Here’s an example output line from ‘battery short’:

G4:~ kord$ ./scripts/battery short
2007-10-07 18:34:27 1 _________i__ 11.232V -1.454A 2.788Ah of 4.720Ah (59.1%) of 4.400Ah (107.3%) 13 cycles

The line of underscores with an ‘i’ in it are the battery flags set. ‘i’ means my battery is installed. Duh. Other flags include whether the lid is closed, the battery is on fire, or it’s just on the charger. See the battery.rtf file for more information on the flags. I have a G4 laptop, but just got my battery replaced for free! Only 13 cycles on it so far!

Splunk eats logfiles, so you’ll need to get a logfile rolling on your battery output. I’m going to assume you know how to use vi (text editor) do the rest of this work.

You’ll need to set up a cronjob to create the logfile and continue logging to it every so often. Switch to root and create a logfile for battery in /var/log:


G4:~ kord$ su
Password:
G4:/Users/kord root# cd /var/log
G4:/var/log root# touch battery.log
G4:/var/log root# chown kord battery.log
G4:/var/log root# ls -la battery.log
-rw-r--r-- 1 kord wheel 0 Oct 7 18:45 battery.log
G4:/var/log root# exit
G4:~ kord$


Now use ‘crontab -e’ and put in a line that looks something like the second line of this:


G4:~ kord$ crontab -l
* * * * * /Users/kord/scripts/battery short >> /var/log/battery.log


That will cause the battery script to run once a minute and append it to the battery.log file in the log directory. After a few minutes tail the logfile with ‘tail /var/log/battery.log’ and make sure you’ve got data in there. Also, I’ve edited my own crontab, but you could elect to do it as root (thus skipping the chown step above).

Obviously you will need Splunk installed to chart the battery usage out of the logfiles. If you haven’t installed it already, there’s a free version up on the website you can download. Follow the instructions for installing it on OSX.

Assuming that you installed Splunk in in ‘/Applications/splunk/’ you can do the following to start it:


G4:~ root# cd /Applications/splunk
G4:/Applications/splunk root# export SPLUNK_HOME='/Applications/splunk/'
G4:/Applications/splunk root# ./bin/splunk start


Now you’ll need to download my addon for Splunk, which is basically a bundle of configuration files. For reference, I also put the battery script in the tar file, along with an example crontab file. To get the bundle in the right place, start by un-taring it:


G4:~ kord$ tar xvfz battery.tar.gz
battery/
battery/addon.conf
battery/bin/
battery/bin/battery
battery/bin/battery.rtf
battery/bin/crontab.example
battery/props.conf
battery/screenshot.jpg
battery/transforms.conf


Now move it to the correct location in Splunk’s directory:


G4:~ kord$ su
Password:
G4:/Users/kord root# mv battery /Applications/splunk/etc/bundles/


And restart Splunk now:


G4:/Users/kord root# /Applications/splunk/bin/splunk restart


We’ll spend the rest of our time in a browser, using Splunk’s kick-ass web interface.

If you left the default port alone, you should be able to fire up Firefox and hit http://localhost:8000 and see the initial login screen (or not if you are using the free version). I’ll leave the particulars of getting to the initial search interface on Splunk to you.

Add the battery.log file to the list of files Splunk monitors. Click on ‘admin’, then click on the ‘data inputs’ tab. Click on the ‘Add input’ link to the right of ‘Files & Directories’ at the bottom. Leave the data access to ‘tail’ and give the full path to the logfile - ‘/var/log/battery.log’ in my example above. Host can be constant, DNS name doesn’t matter, and set the source type pulldown to ‘_battery’. Remember, this sourcetype won’t be in the list until you install the battery bundle.

Click on ‘add’ to add the source type. Go get a cup of coffee while Splunk eats this and other files on your computer and builds the index.

Back from the caffeine, you should now click on the ’splunk>’ logo at the top left. Type in the following in the search bar, sans the quotes: ’source::/var/log/battery.log’. Click on the ‘fields’ pulldown on the left and check a few extracted fields, such as battery_ah_remaining, battery_draw, battery_percent, and battery_volts. Click on ‘fields’ again to close and reload with the extracted fields showing.

You should get something that looks like this:

If you have about an hour’s or so data logged, try entering ’source::/var/log/battery.log | timechart avg(battery_draw)’ in the search box at the top to generate a report for the last 60 minutes.

Here’s what my amp draw looks like for the last 3 hours:

The move ‘up’ in the graph halfway through is actually a drop in amps drawn on the battery when I restarted Firefox. The cause? Firefox had a Flash game running in another tab, and it had eventually heated up the processor enough to kick on the fans!

Here’s another one, showing the evidence of me having a newer battery installed - almost five hours of continuous usage after 4PM, with only a few screen sleeps:

It’s interesting how the laptop charges at a rate almost the same as it discharges. It preserves battery life doing it that way, especially with the new lithium-polymer batteries.

See what else you can dig up about your battery. Try charting with some of the flags that are set - like how often the charger is on the laptop, or what the draw rate is if you have the screen clamshell closed.

More video formats are available from Splunk’s Tech Talks section.