Thanks to more testing i have found and fixed a few critical bugs.
Updated APP version 1.6 >> here <<

• there was a static var preventing the multiple server configs from working. Should be fixed, and multiple servers in the vmware.conf should work.
• Ibm jvm’s should work - ie AIX should now work
• Added new saved searches and a few dashboards ( thanks to raffy

As usual, please let me know if you find any bugs.
I’ll type up some notes on my VMworld experince

Cheers,
e

**** UPDATE - 09/08/08 ****
Thanks to lots of folks trying it out i have found a critical bug that was preventing much of the data from getting indexed. This latest release 1.5 should have that fix and everyone should see all the wonderful VMWare data in the index.

As usual, bug me if it does not work or you have any questions.

If you have made changes to vmware/local/vmware.conf and not to the file in default you can just untar this version on top of your old one. If you are making changes to the default/vmware.conf file, i’d move that to local/vmware.conf that way when i ship updates it will not blow away your conf changes. We ship only default and not local/vmware.conf.

Thanks again to everyone that helped find bugs!

e.

**** UPDATE - 08/27/08 ****
I have updated the app with a few fixes found in the field.

• hopefully fixed issue on AIX (IBM jvm )
• added output of host/vm name on update messages. It was hard to tell where the messages were coming from
• added more debugging infor on startup to help debug connection issues.

Things that are still under-investigation.

• Pointing at lots of ESX servers and not VC. Seems as though some data is not coming back from ESX.
• Making work with older jvm’s ( currently it seems i require 1.5)

**** Original Post 08/10/08 ****
I’ve wanted to release this a few months ago but the project keeps getting stuck on the back-burner. Finally I’ve cleaned it up and had a few people try it and it seems to work well. I’m sure there are configurations and versions out there that will have issues - please write me back ( my first name at splunk.com ) if it does not work as advertised.

Reading the below makes it sounds more difficult that it really is. Just download, un-zip, change the server url, username and password in the vmware.conf file, restart and go! This really is the first pubic release and i’d love to get more feedback. I’ll more than gladly send you Splunk tee shirt of your choice if you help find bugs or have useful suggestions!

Why you want to give it a try:
This vmware app is a cool way to keep track of what your VC and ESX servers are up to, what instances are running where, when they are under load, when instances move, when they have errors, and much more. Since all the data is indexed in Splunk, it’s easy and quick to search for problems and report on your virtual sprawl.

How it works:
This app will connect a splunk server to any number of Virtual Center and/or ESX servers and grab/index the events, logs, properties, performance data, and anything else I can get my grubby mitts on. It’s easy to hookup and get going, so if you use Virtual Center or ESX than give this app a try. I’ll explain how to install/setup, how to trouble shoot, and what you will see when you get it working. You will need to install splunk or use an existing Splunk server. See the configuration file for settings on how often to pull data. Also near the end of this post i give example searches to explain the data.

After installing you get cool graphs like this one showing CPU Usage by Guest by Time:

Add Inside-out monitoring
Its optional but if you can also put splunk on the guest OS’s as light weight forwarders and you will get a brilliant inside out view where we capture not only what VC/ESX thinks but what the guests are seeing on the inside. My best practice is to put splunk on the guests and capture basic logs as well as OS performance metrics, what apps are running, how much mem/cpu they are taking, etc. You can get the Unix/Linux version here and the windows here. Of course its not required and you get a ton of value out of just with the basic vmware app’s monitoring of VC/ESX.

INSTALLATION:

**Important**
This app requires a JVM be installed on the same box as the splunk server. I know this is less that optimal. Please bug your local VMWare rep and tell them to make me REST API’s and not SOAP API’s. The VMware API’s are hideously over complicated - Please dear VMware make a simple REST interface.

1) Make sure java is present and set the JAVAHOME environment variable. If not already set you must be set JAVAHOME to the directory that contains the java binary.

2) To test the variable is set correctly, try and run the following on the command line
windows> "%JAVAHOME%\bin\java
linux/unix> $JAVAHOME/bin/java

If it worked it should spit back a bunch of options to pass to the java command. If its not set right you will get some kind of file not found error.

3) Grab the vmware.zip file HERE.

4) Unzip the file - and copy the resultant “vmware” directory to your SPLUNK_HOME/etc/apps/ directory. When done the following directory should exist: SPLUNK_HOME/etc/apps/vmware.

CONFIGURATION:
There are a few config settings to make the app work.

5) First you need to let Splunk know where your VC or ESC servers are. Edit the vmware/default/vmware.conf configuration file to point to your vc or esx servers. If using VC you need not specify all ESX servers under management, splunk will get the list from VC. The config file contains one or more of the following stanza’s ( the unique_name can be anything you like so long as its unique):
[vmserver:unique_name]

For each [vmserver] stanza be sure to set:
url=https://your_server_IP/sdk
username= your_user
password=your_passowrd


Note that the url should be the ipaddr of your server with “/sdk” at the end - for example “url=https://10.1.1.35/sdk”. A good way to test that the url and username/password are correct is test using a web browser. Take the url you have entered above and replace the “sdk” with “mob”. Use the web browser to navigate to that url and make sure it asks for username and password and that the values you entered above will authenticate correctly. If the “mob” url works with the username and passowrd you entered than splunk should have no trouble.

With those three set you should be up and running after a restart.
The rest of the config file should be self explanatory and is included end of this post for reference but you should not need to change anything else.

Testing and Troubleshooting:

6) It’s best to test running the vmware app outside of splunk first.
You’ll need to make sure that SPLUNK_HOME is set for the test.

** On Windows **:
set SPLUNK_HOME=your splunk directory
#note it does not like it when i add quotes around this path - try with no quotes.

Then run the app by hand
> cd %SPLUNK_HOME%\etc\apps\vmware
> java -jar lib/splunk.jar

** On others ** :
export SPLUNK_HOME=your splunk directory

Then run the app by hand:
> cd $SPLUNK_HOME/etc/apps/vmware
> java -jar lib/splunk.jar

It should spit out all sorts of vmware data. If it throws an error its likely that SPLUNK_HOME or JAVAHOME are NOT set. Remember SPLUNK_HOME will be set by the server when the server runs the script. You need only set it for testing.

If it does not work, likely the exception will have something useful in it such as connection refused ( bad auth ) or a 404 error in which case the url is incorrect.

If you get any non-obvious errors email me ( my first name at splunk.com ).

7) Try running in splunk.
If the above test works than you should be able to just restart splunk and all should be good. The way to tell if its working is that you will get events with sourcetype vmware and vmware_api.

8 ) If you do NOT see events of type vmware_api on the dashboard than try the following search:
“index=_internal error”
and
“index=_internal splunk4vmi.py”

You should see some kind of error or warning that is hopefully obvious. If not again email me and i’ll sort you out.

Using the App

At this point it should be working and you should be able to search for cool stuff.
Here is a quick overview of what splunk is indexing:

After restarting you should see a bunch of logs from vwmare and at least two new sourcetypes ; vmware and vmware_api. Below is a screen shot of my dashboard after restarting - notice the vmware logs and the vmware_api event counts.

The vmware sourcetype is for the actual vwmware logs while the vmware_api sourcetype is for the API calls. It can take a minute before they show up so if they are not there, try again after a minute. If you still do not have the logs that likely means the logs path in the vmware.conf if incorrect and you should make sure the path is correct or contact me.

If you do not see the API calls than there is likely an auth or url error that should have been caught when you did the manual test above. Try retesting by hand above - if the by-hand method works but not through splunk than contact me.

I’ve just started to explore the logs that come back - there is a ton of information in them but my test infrastructure is not all that insteresting so i’m not sure what goodness you all might find in them. Poke around the files and see what you see and bug me if you see anything interesting i can make them into alerts / reports.

The meat of the data is from the API where we pull everything we can.
Most useful are:

1) Metrics
Every few seconds we captures the metrics for all VM’s, including

2) Events
I’m not sure the scope of these but it looks like interesting events kicked out by ESX. Someone with a larger VMware installation might find far more interesting events than i see on our infrastructure.

3) Updates:
It looks like when anything changes, we can an update.

4) Inventory:
I periodically just capture the inventory tree. It’s more for debugging than perhaps useful in a production environment but it does not cost much to get and it can be useful.

Thanks to Christina we do ship with a bunch of saved searches. After installing you should see them, they all start with ‘VM:’. They are named to be somewhat obvious, again let me know if they dont work or you have some better ones to add to the default app. Try some of the Metrics and Status saved searches to make sure your install is working.

• VM: Investigation CPU load on all guests sharing ESX server
• VM: Investigation Find ESX Host for Guest
• VM: Investigation Find Guests sharing ESX Server - Non FQDN
• VM: Investigation- Find other VMs sharing ESX Host
• VM: Investigation- Processes on hosts sharing ESX Server
• VM: Investigation- Running processes on other guests on same ESX server
• VM: Metrics- CPU by Guest last 60 minutes * VM: Metrics- Host Memory Usage last 15 minutes
• VM: Metrics- Host Memory Usage last 60 minutes
• VM: Metrics- Memory by Guest last 60 minutes
• VM: Status- Free Space by Datastore
• VM: Status- Running Guests
• VM: Status- Running VMs

That’s about it.
Like i said, PLEASE email me if you have bugs or suggestions.
I’ll plan on updating the app with whatever feedback i get from folks. So please, help me out and get yourself a tee shirt.

Kind Regards,
e.

P.S. - there is a sample of the config just so that you can see what’s in it without downloading:
—————————————
The following are the important values in the config file:


[vmserver:demo]
url=https://10.2.1.151/sdk ## This is the url to the vc or esx server
username=your_username ## user name to auth against the server. If you are not sure of its value point we browser at the above url and check the web auth, it will be the same.
password=your_passowrd ## we will support non-clear text in the near future.
ignorecert = t ## for now leave as true (t), we will soon support checking of certs
loggingLevel = error ## to turn on debugging values are [error, warn, info, debug ]

index_logs = t ## should we index logs (t)rue or (f)alse
logs_interval = 300 ## how often to get log changes…
logs_localpath = ../var/spool/vmware ## the logs are copied from vc/esx to the this directory where splunk will pick them up for indexing


Around here is synonomys with filing 34 bugs between sunday 9PM when we push bits to the site and 9AM when we get in to the office. I dont mean the usual the UI-is-off-by-10-pixels but complex indexing or distributed search bugs. Well, sometimes is its a trivial thing we missed, but usually he is usually pushing splunk to its limits. Its not often that a CTO and “industry expert” is the one to personally put splunk through its paces - but it’s RJ is like that and gets his hands dirty - and splunk is the better for it.

RJ and Voxeo are one of a few, but quickly growing, number of companies that are using splunk in a multi-tenant environment. This means using splunk to to collect data across multiple tenants in a hosted environment and then using splunk for searching and reporting on a per customer basis. Often the output of the searches/reports is rendered for the customer do they can see what is going on within the service. Customer dashboards and activity reports are a common usecase for splunk. Below are some of the images from the voxeo service:

On the Voxeo blog there is a nice description and even a cool video introduction:

Lessons learned from these initial deployments are having a significant effect on our upcoming 4.0 release. First and foremost we will provide a much better html “module” system so that you can embed splunk modules in other webpages. Secondly, we will be having the overall splunk UI more configurable and modular so that multi-tenant customers can build even more custom UI’s.

One other very interesting trend is using splunk for SaS using cloud services. Often these uses have some kind of multi-tenant …. It wont be long before splunk makes deploying in the cloud even easier. More in a post to come but do drop me aline if you want to use splunk in the cloud and i can give you some hints.

In the mean time if your looking for the best push-it-to-the-limits beta tester, contact RJ!
Thanks RJ

e.

I just wanted to congratulate Nathan over at FlowingData for crossing the 3100 subscriber mark.

FlowingData is a fantastic example of the hidden value in the data all around us. As more and more of what we do is documented by computers the impact of statistics has become less of a hard-core math geek sport and more within the reach of anyone’s curiosity. His daily posts are a constant reminder of how statistics has become a crossover genre.

Thank you Nathan!
e

Thanks
e.

I’m the manager of the search and indexing team at Splunk. We’re still in the process of writing up our findings from storage benchmarks but here are the general details.

High IO/s typically means both faster indexing in general and faster searching of rare, temporally incoherent events. On average, we’ve seen indexing speeds increase by about 66% going from an 7200 RPM SATA RAID to a 15K RPM SCSI RAID. We’ve seen comparable performance from SCSI and SAS RAIDs, provided they’re 15K RPM.

The best best benchmarking tool we’ve found for measuring how Splunk will behave on your disk hardware is bonnie++. If your disk subsystem can sustain 800 IO/s, you’re in good shape.

As far as searching goes, IO/s is the dominant factor for non-coherent, infrequently accessed search results. This means, if you’re just searching for the newest data, or even have to reach back through 1MM events to return 10k, the disk is NOT the bottleneck, since each individual read() will pull many events off disk. However, if you’re searching for a rare term, like a name, that occurs once an hour or once a day, each read() is going to require the drive arm move. If you’re using a 7200 RPM SATA drive, that’s about 100 IO/s and hence on the order of 100 retrieved events per second. If you have a decent RAID, that could be 800 retrieved events per second.

-s

It’s taken longer than we would have liked but our 3rd preview build has been posted.
Get’um here

A bunch of work has gone into windows stability, tons of bugs were fixed, and a bunch of customer requests have been implemented ( we will let you know out of band ). We expect that this release should be more stable, slightly faster, and less buggy.

Left to do, we still have a bunch of IE work, performance improvements, and cleaning up of some features like interactive field extraction and event type discovery.

Its still not production ready so don’t even think of trying it out for real - and there is no guarantee that migration will work from a preview to GA ( we will migrate from 3.1.x to GA but not preview ). Also, don’t run splunk as root - its just not good to do until we run through all our testing.

As always, please send us feedback at splunkpreview@splunk.com or hit us up on IRC (irc.efnet.org #splunk).
The last round of info from Preview #2 was awesome please keep it up!

e.

Its packed with holiday goodness, albeit very raw.

First you will notice we have posted a windows build. Its been in the cooker since last Feb and thanks to Mitch, Ledio, Igor and a bit of Amrit we now have a single code base that rocks on linux, mac, solaris, freebsd, aix, AND windows. This was not an easy feat as evidenced by our gift of a pony (soft and electronic) to Mitch for his effort. Its still very raw (the build not the pony), and has a tendency to crash because of a memory fragentation and limited vm space. Which will be fixed by GA… MarkB. will post more on the build so stay tuned for details. Its a big deal for us so be patient and we sure could use feedback on how to make it the best it can be.

Also in this release you will see the UI starts to get some of the async search results. Over the next few releases we will be moving to fully async search in the UI. It will take a few turns but this preview has some of the first cut.

There are a bunch of other improvements; scheduled searches got a bit of a cleanup in the UI and the backend has been improved as well. Performance, bugs, and other tweaks are also spread throughout. I’ll get others to post specifics.

In the mean time, as always its a huge help to us in dev if you can kick the tires before we freeze for GA. Please send feedback to splunkpreview@splunk.com, post comments to this blog, or drop by and tell us in person.

Again, thanks for the help and happy new year from all of us in dev@splunk!

We have posted the our first of many preview releases. You can find them here:
Our hope is that every week or two as new features or API’s become usable that we post builds soliciting feedback.
This first post has a bunch of backend and UI performance improvements as well as some new but hidden features:

• live searching of data
• flexible roles
• scripted authentication
• event decoration ( for the xmas season )
• auditing of splunk server actions
• file system change detection
• improved (proper) sub second support
• transaction search
• new experimental simple search interface
• “where” support in search clause ( you dont need to use the “| where” anymore and can just search for foo=10 )

I’m not going to explain here what these things mean or how to find them or use them
Instead the product managers and developers will post here with ideas on what to try and what feedback we are looking for.

I’d like to thank in advance those brave few of you that have the few minutes to install these builds and give us your feedback.

e.

Just a heads up that we are moving to a model where we post previews of upcoming releases.

Starting now, we are going into a mode where long before a GA release we will be posting development builds. At first, they may be a few weeks apart but over time our goal is to post builds as soon as new functionality or API’s are ready for comment.

This first Preview #1 will have backend performance and scale improvements as well as some cool new features. The developers and PM’s will be posting to this blog the specifics of what is new, how to try it, and where we are going.

Our hope is that we get early feedback on new features and API’s before we actually ship.

Thanks in advance for helping try out our early wares.

Kinds Regards,

e.

If I have a search/report that I want to run faster, I will save that search and have splunk run it over a small timeframe (5,15,30,60 min) taking the results of that search/report and feeding them back into an index i create to hold cached results.

For example, suppose I like to run nightly reports where I show “top users by bandwidth”. Its easy enough to run the report every night, but suppose there are times during the day when I want incrementals, or I want to look at last week, or perhaps get dailies over a month. Every time I run the search/report I need to search and recalculate “top users by bandwidth”, which if over billions of events can take time

Instead, I’ll just save the search/report and have Splunk run it every 15 minutes with the results being sent to a “cache” index. This way if I ever want to do an adhoc search on “top users” or if I want to do “weekly reports by day” all the data is precalculated.

Think of this as creating “logs” that are the output of a search/report and then having Splunk index those “logs”. To get fast results you can then search/report on the summarized cached data.

If not obvious why it’s faster, suppose you are indexing 500M events a day and 100M of those have bandwidth data. To report on “top bandwidth by users” I need to run a search to get the 100M events then run the report across all 100M.
If instead I were in the background running that same search/report over each hour interval, then saving the data back into splunk, I would reduce the data i’m operating on from 100M down to 1200 ( 24*500 ) (assuming that i’m getting top 500). Doing searches/reports on the later dataset are sub second versus the few minutes it would take to run across the 100M.

Make sense ? - its really simple but odd to explain.

PART ONE - Setup:

• 1. Grab the reportcache search script from “** here ** and put it in your SPLUNK_HOME/etc/searchscripts directory - no need to restart you can now cache any search/report data.
• 2. Add a cache index - either add the following to your etc/bundles/local/indexes.conf or create a new bundle and add to that indexes.conf You will need to restart splunk after adding the index.
  
[cache]
homePath = $SPLUNK_DB/cache/db
coldPath = $SPLUNK_DB/cache/colddb
thawedPath = $SPLUNK_DB/cache/thaweddb


PART TWO - Testing by writing to a file:

I recommend that you first test reportcache by having it output to a file that you scan to make sure things look right.

• 1. Find a search you want to cache. Simple candidate is something like the following report against the internal index that shows queue sizes by queue name.
  index=_internal metrics "group=queue" timechart avg(current_size) by name
• 2. Once you have a search you want to cache - add the following "reportcache index=cache path=/tmp file=testcache.log notimestamp" command to the end. The following assumes you have made an index named “cache”. The index attribute is required and you should not use your default unless you know what your doing. Also we are going to output the file to /tmp/testcache.log using the file and path attributes. The notimestamp option simply suppresses adding a timestamp to the filename.
  index=_internal metrics "group=queue" | timechart avg(current_size) by name | reportcache index=cache path=/tmp file=testcache.log notimestamp
• 3. Run the search and you should get back the normal search results and not see an error on the screen. If you do see an error it should be self explanatory.
• 4. Open the file /tmp/testcache.log and make sure the results look ok. They should look like a bunch of lines key=value, key=value

PART THREE - Writing to an index:

• 1. We are now going to have the command put the results into the index. Simply remove the file, path and notimestamp attributes
  index=_internal metrics "group=queue" | timechart avg(current_size) by name | reportcache index=cache
• 2. Run the command - you should again see normal results and no error.
• 4. Wait 30 seconds or so…
• 5. Run the following search to make sure results made it into the cache index - you should see your cache data after this search
  index=cache
• 6. Now click on the report link and see if you can get your report back This part is the somewhat odd part. All the fields should be as they were in the original search but many reports create keys with odd names. The best thing to do is to click around and see what reports you can make. You should be able to get back to the original search/report prior to the caching.

PART FOUR - Enabling automatic caching:

After you have found and tested a search/report you want to cache moving forward:

• 1. Save the search along with the reportcache command
• 2. Schedule the saved search on a small time frame ( 5, 15, 30, etc ) minutes
• 3. Test by waiting a few hours and looking at the results in the cache index.

There is a good chance that either the above description was vague or that there is a bug / edge-case that i did not consider.
One frequent problem i have seen is trying to cache data that has no timestamp. For example,
somesearch | top users
will produce restults without timestamps. This makes a mess of the cached data. If you have this problem then try rewriting your search to something like:
somesearch | stats count first(_time) by users | where users != "" | sort -count
The above will produce data that has both top and timestamps.

Few other things that are common requests:
Often folks want to go back in time and create cached results for prior data. I have a script that can do that and will post it after more testing.
Another common topic of conversation surrounds the over creation of summary data. In many cases it can be benificial to cache more stuff than you initally need in case you want to run reports later. I’m trying to think of good ways to automatically do this for you.

** IMPORTANT ** - drop me a line and let me know how something like this *should* work. I suspect that we will add a “checkbox” to saved searches that will automatically do the right thing.

I’ll leave this post wit the usage info from the top of the search script.

# usage: | reportcache
#
# file=[filename] - default is current time
# path=[path] - default is $SPLUNK_HOME/var/spool/splunk
# index=[indexname] - which index to target for results. If blank will use whatever is bundled
# marker=[string] - this is just a token or k=v used to mark the results for version or other delination or to defeat crc caching
# format=["csv"|"splunk"] - use the output format “splunk” for feedking back into splunk or csv if you want to save for other tool
# appendtime - if true this will append current time. Its useful when you are doing something that you want with a timestamp of now
# notimestampe - is this arg is supplied it will suppress the timestamp in the filename
# debug - if debug then will just out args to screen

# following example will put in var/spool/splunk a file named foo without timestamp, marked with erik=nextrun”, and targed to index cache
# index::_internal “group=pipeline” | timechart avg(executes) | cacher file=”foo” notimestamp marker=”erik=nextrun” index=”cache”