Dont forget to index your config files!

Why?

Because splunk is a great way to track changes and see differences in your configs.
For most troubleshooting and compliance situations having a historical recored of all your configurations just goes hand in hand with the log data. They are two sides of the same coin.

The cool thing is that it takes just a few seconds to get up and running. If you have splunk installed its all but free to index your configs - they are small in size compared to log files. Even if you indexed all configs in a 2000 machine deployment it would not come close to the volume of even a small size proxy log.

30 second refresher:
Just tail /etc you will capture most of the interesting configs on your box.

from the cli:
> splunk add tail /etc

or in UI just add a tail to /etc

Thats it. That is all you need to do.

** note ** you should grab 3.1 ( http://download.splunk.com ) as there were some bugs in 3.0’s config processing.

A standard preamble about life at splunk.

We are always looking for passionate and intelligent engineers regardless of their background, musical preference, grades they got in collage, or ability to prove some esoteric math theorem. We react best to people that are creative and think for themselves - people that are smarter than shit but don’t over think the wrong thing.

Much of our companies identity, the product, its features, how its used, how we talk about the product, our branding, all mainly come from the development staff - and we plan to keep it that way. Our philosophy is based on the idea that 10 diverse smart people in a room are better off than 25 decent yet uninspired engineers - so although we need to grow fast we end up going slow because we are picky.

Couple of other data-points:

• we like to have fun while at work
• we have smart people, challenging problems, and an interesting architecture
• we practice an liberal interpretation of scrum
• our code is cross platform and *resembles* open source development models
• we like nice hardware ( cpu, monitor, etc ) and dont care what OS you use