Best Practices
Part of my focus as Knowledge Manager at Splunk is capturing best practices, particularly the best practices around using Splunk. Gathering this information is a lot like researching a story, getting me to draw on:
- My own knowledge of how to use Splunk (hey, I’m not a guru but I have been using it for over a year now)
- Our sales engineers who encounter what customers want to do in the field
- Folks in support who hear a lot about what aspects of Splunk might be confusing, along with what folks are trying to do with it
- Our developers, who can give me those sweet little tips and tricks that I can share to help people do things more efficiently
- The folks in professional services, who definitely encounter what customers are doing in the field.
- Doc writers, who spend a lot of time explaining the nitty gritty of things and can often have some pretty interesting tips.
- And really anyone else in the company who might have something interesting to share.
If you get the feeling that it really means talking to everyone at Splunk, it practically does. Knowledge Management, after all, basically involves pulling all of our internal expertise out and putting it into a central form where it’s useful to us and to our users.
The fun part of this, to me, is that it’s resulting in a huge pile of really interesting documents. (Yes, I said fun.) Most of these are technical how-tos and best practices writeups as I look into not only how to do things, but the best/most efficient ways to do things. I look forward to publicly sharing the first set of these sometime in the next month. In the meantime, if you have any thoughts, tips, tricks, or questions, feel free to toss them my way or share them on our forums.
March 21st, 2008 at 1:44 pm
Hi Dee,
Here’s a tip I’ve picked up the hard way. An enterprise deployment of Splunk is likely to use the Distributed Splunk features, forwarding, receiving, and indexing across multiple Splunk servers. It is also nice to have (near) real time access to your data. In order to get the best performance out of a Distributed environment, it helps to have your log files explicitly tailed by Splunk on the forwarding side. You can do this with the CLI by saying splunk add tail /var/log/messages, or similar.
An alternative that works less in real time is to simply tail the /var/log directory. The problem here is that Splunk has a tendency to read files for a while, then go away and come back later. This increase in lag can be frustrating. The explicit tail is preferred in this setup.
Hope that is useful to someone out there.
-D
March 26th, 2008 at 4:13 pm
Thanks for your comment Demetri.
So you find it works better to list each file than to just tail a whole directory? Interesting.