What’s going on with AWS and Splunk…

All of my posts seem to be sparked by some sort of customer interaction.  The last few have been about how to do something, but this one is about what we are doing.  A customer recently asked:  ”What can you do with Splunk in AWS?”.  While there are some docs and posts that cover different tasks, there isn’t a single place to get a wide view of things.   So, here goes….

First, Splunk software will happily run in EC2.  As it is software that is agnostic to the operating system and hardware, EC2 is a fast way to get started with Splunk.  To provide some comfort on the topic, we test and run our own Splunk instances on it. …

» Continue reading

Network Inputs – Best Practices…

When architecting a Splunk deployment, there is almost always a requirement to support syslog event streams from many devices. While Splunk can easily accept syslog data directly from these external devices, you may be wondering if there are best practices around this.  For you long time Splunk users, this should be old news and possibly a refresher.  For you new Splunk users, read on…

So what do the experts do with Network Inputs?

First, I’ll defer you to another post about deciding when to use a Forwarder to route the data into Splunk: http://blogs.splunk.com/2011/10/24/choosing-a-forwarder-or-not/.  There are important concepts in that blog that will help decide your best setup with respect to forwarders.  So let’s think about the layers outside …

» Continue reading

Tips and Tricks for the new guy

Before we dive into the meat of things, let me first explain how this came up…   A newer user/admin of Splunk was attending our conference (#datajourney) and found that there was this convenient command that checked configs.   This command is called btool and it has been in the product for some time.  Now, to me this is old stuff but to him this is very very useful as a newer Splunk admin/user.  It may also build your street cred with the long time users. Well, to his request, I’ll share some of the these tips in this post…  Some of these are CLI commands (noted by ./splunk) and the rest are searches. Here we go…

Checking what Splunk

» Continue reading

Best Practices with Splunk

So it is that time of year again, where all the Splunkers unite in one location to talk Splunk.  This means the best and brightest that work for Splunk will be out in force, ready to talk and teach you about the best ways to work with it.   I’m the lucky one that gets to talk about best practices for virtualization/cloud deployments, as well as fundamentals of architecting and sizing.  People from services, product management, and engineering will be there to talk and show you how to do it right…or maybe how something new can make life easy.  For all of you long time Splunk users and administrators, this means there is awesome new content for you to enjoy.…

» Continue reading

Splunk and AWS sizing revisited

Some time last year, I posted some recommendations for running Splunk on Amazon Web Services (AWS).  While the base recommendations for how to size and architect Splunk have not changed, we do have more clarity into what works best.  Instead of editing that post, I decided that it would be best to review the thought process and give more color to what most people are doing with it.  Before going down the road of sizing on EC2, I highly recommend reviewing our standard documentation.

For general sizing purposes, there are two key factors:

  1. Daily Indexed Volume (how many GB indexed per day?)
  2. Searching and Reporting needs (how many searches or alerts will be run?)

Most people will already know …

» Continue reading

Restoring an index

In a recent post, I covered some details around a backup strategy.  I left a bit of a teaser at the end, stating I would follow up with a post on index restoration.   Well, here it is…

There are a few scenarios you may encounter when trying to restore or recover an index.  The simplest scenarios, such as moving an index, are covered very well in the moving indexes wiki topic as well as on our answers site.  From a high level, you can move indexes across Splunk installations but must consider the following:

  • The Splunk instance receiving the index has never been configured with an index of the same name – this prevents bucket ID collision
» Continue reading

Splunk and Chef

For those of you that run Chef in your Splunk environment, or are thinking of doing it, I have some great news. There is now an open source code base on github. Big thanks to Bryan Brandau and Aaron Peterson for working on this! Here is the official tweet and link:

https://twitter.com/#!/agent462/status/154640900566433792

https://github.com/bestbuycom/splunk_cookbook

» Continue reading

Index backup strategy

In this post, I’ll cover one strategy to backup your index.  Before we go any further…

  • Do not do any of this on your production system without testing
  • This applies for version 4.2.x only
  • You should have a very good understanding of Splunk administration, indexes, and buckets (http://docs.splunk.com/Documentation/Splunk/4.2.4/admin/HowSplunkstoresindexes)
  • Read this:  http://docs.splunk.com/Documentation/Splunk/4.2.4/Admin/Backupindexeddata

Let’s assume we have a standalone Splunk deployment that indexes 10 GB/day.  Our goal is to make sure we have a backup on a daily basis, extending all the way back to Splunk’s first received event.   The strategy encompasses a few steps that basically take chunks of the index at set intervals.  We will accept the potential to lose data for the last day, but want to be …

» Continue reading

Choosing a Forwarder, or not

When deploying Splunk in the wild, there is the task of deciding “to forward, or not to forward”.  This decision comes down to many factors, but the typical response/answer is to use the forwarder.  In this blog, I’ll detail that decision process so you can decide for yourself.

First, let’s quickly explain what a Forwarder does…if you already know, skip to the next paragraph.  Splunk can perform four basic functions:   searching, indexing, forwarding, and acting as a deployment server.   When Splunk is setup to be a forwarder, it reads in the raw data and sends it to a Splunk indexer.  In the latest version of Splunk, we offer an additional software package especially for forwarding (only).  This is …

» Continue reading

How can I get 2 days of in person Splunk training?

Come to .conf 2011!!!  The 2nd annual Splunk user’s conference is upon us in a few weeks.   It is hard to describe how much knowledge is spread throughout the Splunk world in such a short period of time.   Attendees get the latest on Splunk, best practices, solutions insight, product direction, and tons of training content.  Our best engineers are presenting really cool content and this is your chance to interact with them.  While Splunk’s best talent is there, our best customers are also there presenting really cool stuff.  I must say after watching some of the customer presentations last year, I came away with new ideas on how to get more value out of Splunk at other customer …

» Continue reading