Universal Forwarders and the Splunk App for Active Directory
About once a week I respond to a call or online question asking about the Splunk App for Active Directory. Specifically, these questions ask one of two things. The first is “can I collect the Active Directory data remotely?,” and the second is “What user shall I run the Universal Forwarder as?” The cliff notes version is that you should not collect Active Directory data remotely, and you should install the Universal Forwarder as the system local user. If you want more information, read on.
Let’s start with the first question – can you collect the Active Directory data remotely? Technically, the answer is yes, but reality is the answer is that it is ill advised from a security …
Which Microsoft Servers are inactive?
What can you tell me about my environment? It’s a common enough query and Splunk seems to be able to answer them all. The latest was this: Can you give me a list of all the servers that are inactive? Inactive, for the purposes of this post, means that they are bound to the domain but they have not logged into the domain in some period of time.
One of my favorite tools for answering these questions is the SA-ldapsearch commands. Fortunately for us, Active Directory contains the timestamp. Unfortunately for us, it contains two timestamps. The first is called “lastLogon” and contains the time stamp that the system in question last connected to THIS domain controller. The …
Measuring Windows Group Policy Logon Performance
One of the common complaints you will hear from Windows users is that their logon takes too long. This is especially true for Microsoft Remote Desktop Services and Citrix infrastructures. Luckily, Microsoft is logging all the nitty-gritty details in Event Logs. So, naturally, Splunk can give you insight into what’s going on.
Getting the Event Log Data Into Splunk
The Windows event log holding this data is found under Applications and Services Logs > Microsoft > Windows > GroupPolicy > Operational.
Splunk has a built-in Windows Event Log collection mechanism, but that mechanism will not get the raw XML data behind the scenes of these entries. We need the extended XML data for our analysis. So, using a little PowerShell …
Forwarding Windows Event Logs to another host
Let’s face it – sometimes, it just isn’t possible to install the Universal Forwarder on all hosts. Mistrust of new software, proof of concepts and security concerns all play into the decision to install a Universal Forwarder or not. What do you do when you can’t install a Universal Forwarder? In this article, we will discuss how to configure a Microsoft Windows host to forward the Windows Event Logs somewhere else.
Throughout this article, we will refer to the “source” when we mean the system that is generating the logs in the first place, and we will refer to the “collector” when we mean the system where you are centralizing the logs.
Step 1: Configure WinRM
Your first step will …
Working with Active Directory on Splunk Universal Forwarders
Have you ever installed a Splunk Universal Forwarder and seen one or more of your Active Directory domain controllers have high CPU utilization as a result? Have you ever wondered how the Splunk Universal Forwarder translates the Security ID effortlessly into a real name you can read? In this blog post, I’m going to tell you exactly how we do the things we do with Active Directory and how you can improve the performance or reduce the load on your domain controllers.
There are two Windows pieces on the Universal Forwarder that deal with Active Directory. The first is known as admon – it emits information about your Active Directory Domain Services objects – both as a “dump” of the …
Active Directory Replication and Windows Server 2012 R2
If you have upgraded your Active Directory domain to Windows Server 2012 R2 and use the Splunk App for Active Directory, you may have noticed that the replication statistics script doesn’t work the same way as on older versions of Windows. Specifically, the ad-repl-stats.ps1 script takes forever to run and consumes just about as much memory as you can give it. This is because of a change in the implementation of the System.DirectoryServices.ActiveDirectory API that Microsoft provides. In prior releases of Windows Server, the API was lazy – data was only filled in within the objects when the data was requested. In Windows Server 2012 R2, those same objects filled in the data at instantiation. When we read the …
Logging DMVs from Microsoft SQL Server with PowerShell
Some systems are easy to monitor and diagnose – just Splunk the log file or performance counter and you are pretty much done. Others take a little more work. Take, for example, Microsoft SQL Server. Many of the best bits of management information are stored in Dynamic Management Views, or DMVs. Getting to them is not so straight forward.
In order to get those nuggets, we need to do some pre-work. Firstly, install a Splunk Universal Forwarder on the SQL Server. Then fire up the SQL Server Management Studio and add the LOCAL SYSTEM account to the sysadmin role. This will allow the local machine access to all the information you need to monitor any database within the instances …
Monitor Processes Per User on Microsoft Remote Desktop Services Session Host
Microsoft Windows Remote Desktop Session Host (formerly Terminal Services) hosts multiple users on the same Windows Server Operating System. Therefore, all these users are sharing the same resources available to the OS. A lot of administrators want to know which processes belong to which user and how much resource allocation is used by each of these processes. This way, it is possible to determine power users or application resource hogs. This is useful in RDSH-based environments such as Citrix XenApp, Dell vWorkspace, Ericom, and more.
Step 1 – Index Process Information
The first thing to do is gather all the running processes and the desired metrics via inputs.conf. Here is a sample I use:
Detecting Attachments in Microsoft Exchange Server 2013
One of the common recurring themes I get is how to detect attachments and log those attachments in Splunk. Let me get the obvious piece of this out of the way first – you cannot log the attachment names or contents without a Transport Agent. This is a special piece of code that is deployed on all your Exchange Servers that intercepts the messages as they go through the system and does something to them. You will normally see a transport agent deployed for anti-virus scanning, for example.
However, logging the fact that there is an attachment is relatively easy. You can create a Transport Rule to log a message when attachments are created. To create the rule, you …
Decoding IIS Logs
Everyone (just about) knows that there is a table of status codes that HTTP/1.1 defines. However, IIS gives you two more status codes in the log files. The HTTP/1.1 status is stored in sc_status (and it is automagically decoded for you in Splunk 6). There is also an extended code called sc_substatus and a Win32 error code. How can you really decode these, especially since the sc_win32_status seems to have really large numbers?
Let’s start with the sc_status and sc_substatus codes. These are normally written together as a decimal number. So, for instance, 401.1 means an sc_status of 401 and an sc_substatus of 1. The sc_status codes follow a pattern: 1xx are informational, 2xx indicate success, 3xx indicate redirection, and …