I’ve got another post here in the occasional “Help Me Help You” series, this time I’m going to digging into case writing.

I was talking with the some of the engineers the other day around the bar about an issue that one of our field guys opened. One of the engineers mentioned a piece of information that totally changed the way the rest of us were going to handle the issue. This got us to talking about how some people write great cases and others don’t. The ones who write good cases usually get their issues resolved first (often times closing the issue with the first response from a member of my team), the ones who write “bad” cases seem to have much of a back and forth exchange.

That got me thinking that maybe I should take a sec to talk about what makes a good case. I’m going to try mapping out a basic template for submitting an issue. This is by no means limited to Splunk and is most definitely not a de facto standard. Rather it is a compilation of things that always make my life easier when my customers can provide them.

• Backstory: Like I mentioned in my previous post I don’t work in the cube next to you, I don’t see the same things you see, know the same things that you know.
  Often times I get cases with a description like “I came into work this morning and discovered that this thingy that was working yesterday isn’t working today. What gives?” In digging into the issue the customer remembers that last night was the weekly maintenance window and one of the other guys was making some changes on the box and it is this change that caused things to go wonky.
  I guess what I am getting at here is that it helps to know what led up to the issue. Flushing out the supporting data points can be a big help in piecing the problem together. Even if you think it is unrelated include it, it can’t hurt. The worst thing that can happen is you spent a few more bits and thankful bits don’t cost what they used to. I’ve also found that when I take the time to think about _all_ of the things that led up to the event in question the light bulb over my head starts to flicker and maybe I can figure it out before enlisting someone else.
• Impact: Do you have to commit seppuku if this issue is not resolved in the next hour? If you do you may want to include that in the initial report, it will really help with prioritizing the issue. Are others unable to do their job because of this, we want to know. If you’re asking a question for your own edification share that as well — helps us to prioritize other issues and formulate the best answer for you. Big fires often require an immediate fix and you don’t really care about the inner workings of the fix just that it works. If you are trying to learn something you want the opposite.
• Priority: We all deal with fires (some bigger than others) let the guy on the other know how you need the issue treated. Support folk inherently want to help (why else do we do this job? It isn’t for the unlimited supplies of handi-snacks) and if you say need this now we will make every effort to deliver.
• Data Samples: One of my new favorite shows is The First 48 which follows real homicide cops as they investigate murders. Each episode always starts off with the cops going to crime scene collecting every potential piece of evidence. They don’t know what is relevant and what is not so they assume it all is. The same is true when troubleshooting an issue with software. The more data points I have to work with the better position I am in to figure out what is going on.
  If splunk isn’t parsing a field in a given file include a copy of said file along with your configs. If the UI is acting weird take a screen shot. If performance is an issue include the results of your tests to determine that things are slow along with the tool(s) used to produce the results.
• Repro steps: If you can trigger this issue on demand, please share. Knowing the exact path traveled will often make root cause analysis that much easier. Screen shots of each step are very helpful (a picture is worth more than a 1,00 words) in describing an issue.
• Your investigation: I find it is really helpful to know what you have done to try to figure out a problem. It saves time because I wont ask you to perform steps that you said you’ve done and you wont get frustrated at me for asking you to do work again. It also gives me insight into your investigative process — if you are thorough I am more inclined to trust your results at first glance. If you are vague or unclear I have to assume that the information you are providing is incomplete. This is not to say that what you are giving is bad/wrong/stupid, rather it is not the full story.

Ok I’m sure there is more that I can say here but this post is getting kind of long, my fingers are tired of typing, and I need to answer some cases.

Wanted to take a minute to talk about authenticating Splunk against Active Directory. In case you didn’t know Active Directory is running on top of LDAP. While the guys up in Redmond do their best to make sure tha you have no need to know LDAP they give you the ability to interface with it over LDAP if you know what you’re doing. Let’s take this time to let you know what you need to do.

If you are comfortable with the command line you can run the command ldifede. The ldifde command is the windows equivalent of ldapsearch and should allow you to get an ldif entry for yourself and a group. With those two entries we should be able to come up with authentication.conf that will allow Splunk to authenticate users.

For those of you that are more comfortable with a GUI The Sysinternals team offers a nice utility called Active Directory Explorer. This gives you tree view of your Active Directory/LDAP structure.

The information provided from these utilities is pretty much everything you need to know in order to follow along with the documentation. If you are still struggling to get it working send an email to support@splunk.com with the output from the ldifde command and your authentication.conf and someone from team will help square you away.

As one of the Splunk Support Monkeys I am going to try to start a semi-regular series of posts on a topic that is near and dear to me — getting the Splunk community to be able to troubleshoot their issues without the need to reach out to the Support Team.

The most important piece of any troubleshooting exercise is getting a solid understanding of the problem. The common statement “Shit is broke” while ’summarizing’ the problem doesn’t do much in the way of isolating the specific problem. Taking a minute or two to think about the problem at and documenting the sequence of events leading up to the problem goes a long way to getting outsiders up to speed on the issue.
Here are few things to keep in mind when working with support:

I don’t work in the next cube over.

This means I don’t have insight into all of the other moving parts of your network. Try avoiding acronyms that are specific to your organization. I don’t know the naming convention that you use for machine names, so if one box is in LA and the other is New York tell me, don’t expect me to know that foo.company.com is sitting in the LA data center.

Less is not more.

You can never give a support engineer to much data. Often times folks think that they have identified the offending error message in the logs and provide that one line in their support ticket. The problem with this is that the support engineer does not get the benefit of context. Most errors are the result of a series of events leading up the final failure. Being able to see what was going on leading up to the problem often times is what allows us to identify cause. The basic rule of thumb is if you think it would be at all useful share. If I can channel Don Rumsfeld for moment: It easy to know what you know, it is hard to know what you don’t know.

Reduce the problem to the fewest number of variables possible.

Remember your 7th grade Algebra class and those complex equations that Mr Buckner had you had solve? You started off solving for x and then you went back using your knowledge of x to determine the value of y. The same is true when troubleshooting software. When you try to solve 4 problems at once you end up polluting your results; you can’t tell if the change you made for x resulted in y blowing up. By breaking the problem into smaller chunks you are operating in a more scientific manner and the results have more credibility.

Log like there is no tomorrow.

Debug logs are your friend. In normal operations the logs don’t need to be verbose but when you are trying to figure something out why not give yourself the benefit of the secret messages that the developer put in the code for precisely this reason. It is also helpful to push the existing log file out of the way when starting in a debug mode. While I said early that you can never give a support engineer to much information the majority of the stuff in your logs (especially if you’ve been running for awhile) is going to be white nows. Starting in debug mode with a fresh log means that the problem and the only the problem are going to be in the log.

As one of the splunkers responsible for answering the phone I’m going to use this space to talk about something near and dear to my hart — empowering my customers so they are able to figure out their own problems thereby allowing me read FARK all day long.

Since we recently released our Windows version a bunch of the folks in the office have been trying to figure out how they do the things they do in a UNIX enviornment (like wget a file) in Windows. I’ve been sharing some of my favorite Windows resources here at the office and figures the rest of you would probably like to know about them as well.

Google
Everyone seems to start here when they are looking for something. Most however don’t know that http://www.google.com/microsoft will restirct your search to Windows sites. They also have these search sites for linux, bsd, and the mac.

SysInternals
Mark and Bryce have created the ultimate coolection of free Windows utilities. Simple executables that allow to get so many of the diagnostic/monitoring things that a UNIX admin takes for granted. Some of my favorites (and especially useful in working with Splunk) in no particular order:

• AccessEnum
  Lets you see who has access to what. This is really helpful when trying to figure out why Splunk isn’t indexing one of your files.
• Process Monitor
  Watch the registry, running process/thread/DLL, and file system usage in real-time
• PS Tools
  A bunch of command-line utilities for listing the processes running, working with the event log, rebooting the machine, etc.
• Active Directory Explorer
  Advanced viewer/editor for Actiive Directory. This will be a godsend you are trying to configure Splunk to authenticate against your domain controller
• WhoIS
  Doesn’t do much in the way of troubleshooting Splunk, but who doesn’t want to be able to see if ultramegaextrmeme.com is available and if not who the lucky owner is? BTW it is available.
• TCPView for Windows
  Lets you see all the TCP and UDP endpoints on your system, including the local and remote addresses and state of TCP connections.

Hope that helps you guys out. All of you experienced Windows folks if you’ve got others out that there post to the comments. If my jaw hits the desk when I click the link I will send you a Splunk koozie.