Active Directory Replication and Windows Server 2012 R2

If you have upgraded your Active Directory domain to Windows Server 2012 R2 and use the Splunk App for Active Directory, you may have noticed that the replication statistics script doesn’t work the same way as on older versions of Windows. Specifically, the ad-repl-stats.ps1 script takes forever to run and consumes just about as much memory as you can give it. This is because of a change in the implementation of the System.DirectoryServices.ActiveDirectory API that Microsoft provides. In prior releases of Windows Server, the API was lazy – data was only filled in within the objects when the data was requested. In Windows Server 2012 R2, those same objects filled in the data at instantiation. When we read the …

» Continue reading

Logging DMVs from Microsoft SQL Server with PowerShell

Some systems are easy to monitor and diagnose – just Splunk the log file or performance counter and you are pretty much done. Others take a little more work. Take, for example, Microsoft SQL Server. Many of the best bits of management information are stored in Dynamic Management Views, or DMVs. Getting to them is not so straight forward.

In order to get those nuggets, we need to do some pre-work. Firstly, install a Splunk Universal Forwarder on the SQL Server. Then fire up the SQL Server Management Studio and add the LOCAL SYSTEM account to the sysadmin role. This will allow the local machine access to all the information you need to monitor any database within the instances …

» Continue reading

Splunking Windows PowerShell Commands

This years user conference was another great conference and we got a ton of questions from you during the conference. Some of them I couldn’t answer at the time – I’m making up for that in between blog posts about new features. The first one was “Is there any way I can splunk what PowerShell commands are being executed on a server?”

There are two pieces of this puzzle: firstly – can I turn on an audit log that includes all the PowerShell commands that are executed within the system? We do that normally through group policy. Open up the group policy management console and take yourself to:

Computer Configuration\Administrative Templates\Windows Components\Windows PowerShell

In this group policy container there is …

» Continue reading

Monitoring Windows File Share Permissions with Splunk and PowerShell

I stopped my last blog post on Windows File Shares noting that there was still more to do. Monitoring Windows File Shares is a three part puzzle:

  1. Accesses
  2. Share Changes
  3. Permission Changes

We have already handled the first two, so this blog post is all about the final one – monitoring permission changes.

Let’s first consider how one would do this generically. As with the file shares, there is a WMI class for monitoring permissions, but it’s harder to use. You need to do it on a per-share basis, like this:

gwmi Win32_LogicalShareSecuritySetting -Filter "Name='$shareName'"

The Win32_LogicalShareSecuritySetting is a complex beast. Fortunately, we only need to know a couple of things. The most important one is the security descriptor. You …

» Continue reading

Monitoring Windows Shares with Splunk and PowerShell

I sometimes get emails after blog posts. One of the (fair) criticisms is that I sometimes do something in PowerShell that can be quite legitimately done via another data input like WMI. While this is true for simple cases, it’s not always true. Take the request, for example, of monitoring network shares. There are three parts to this. Firstly, producing a monitor of the share itself; secondly, producing a monitor of the permissions on the share; and finally, monitoring the file accesses utilizing that share. I’ve already blogged about the last one. Let’s take a look at the first two.

You can actually monitor the share itself using WMI. Network Shares are exposed via a WMI class Win32_Share. …

» Continue reading

Export Search Results with PowerShell

A while back, I wrote an introduction to how you could play with our C# SDK from PowerShell. And just the other day, Adrian wrote a post talking about how you could export really large result sets to CSV, using the REST API. It was a good read, but there was one problem: this was a somewhat Windows-centric post (talking about SharePoint data in his case), but he used curl to get the data out! We can most certainly do better than that for our Windows community, so that’s what I’m here to help solve.

What I ended up doing was to take an example from our dev docs about the search/jobs/export REST endpoint that looks like this:…

» Continue reading

Monitoring Scheduled Tasks with PowerShell

I did the unthinkable yesterday. I combed through my posts for non-spam comments. I apologize to everyone whom I didn’t answer – we get a lot of comment spam that I have to wade through when I do this. However, there were a couple of requests in there for future topics and I’ll try and cover those requests in the next few weeks.

The first request was for monitoring scheduled tasks. I’m going to read this as “given a Windows host, how do you determine what scheduled tasks are enabled and whether they are failing or succeeding?”. That’s a tall order, so I looked to my favorite tool – PowerShell – for the answer.

» Continue reading

PowerShell version 2

By now, you are probably aware that I love PowerShell as a method of getting things on Windows. It’s your one stop method for getting all sorts of nice things. However, our SA-ModularInput-PowerShell module had certain limitations. Most notably, it could only work with .NET 4.5 and CLR4 – aka PowerShell v3. This was great for your one-off scripts where you weren’t adding in any plug-ins. In particular, Microsoft applications such as SharePoint 2010 and Exchange 2007 require PowerShell v2 support because their plug-ins are distributed for .NET Framework 3.5.

I’m happy to announce that one of our PowerShell MVPs – Joel Bennett – has updated the Splunk Addon for Microsoft PowerShell to support .NET 3.5 and CLR 2.

There …

» Continue reading

Splunk Universal Forwarders and the Domain User

One of the things that you have to decide right up front on Windows is how to run the Universal Forwarder. For most situations, running as the Local System account is adequate, providing access to all necessary resources. Other times, you need to run as a domain user; either because of local security policies or because what you are monitoring requires a domain account. For example, SharePoint, SQL Server and remote WMI access all require a domain account. I’ve blogged about how to do the necessary security changes using GPO before, but GPO has some drawbacks. The most notable one is that you cannot have different group policies managing the user rights because the last group policy will overwrite the …

» Continue reading

Detecting Your Hypervisor from within a Windows Guest OS

Let’s face it – most of our applications run on hypervisors – Microsoft Hyper-V, VMware or Citrix XenServer seem to be the top contenders. This makes our technology stacks that much more complex since we have added a layer of abstraction between the application and the bare metal. Instead of a stack that includes Compute Platform, OS, and Application, we’ve added Hypervisor to the mix. How do we correlate what is happening on the compute platform to what is happening on the application level? How do we understand which other applications are running on the same hypervisor?

» Continue reading