Using Splunk to Monitor Changes to PowerShell Scripts

I had a question this morning from a customer who was looking for ways to monitor changes made to PowerShell scripts in their environment. They wanted to know who made the changes, but also what changes were made. Well, I thought to myself–that’s a great excuse for a blog post!

Let’s break this down into two separate requirements:

  1. I want to know when a PowerShell script has been modified
  2. I want to know the changes between two versions of a file that has been modified

Who changed a file and when?

Requirement #1 is not hard to do using Splunk in combination with some native Windows file auditing features. In fact, it’s such a common use case that we’ve documented all the steps in …

» Continue reading

Monitoring Local Administrators on Windows Hosts

It is always gratifying when one of my readers comes to me with a problem. I love challenges. This one had to do with one of my old posts surrounding Local Administrators remotely. Of course, the way to do this is via WMI. However, it doesn’t quite work the same way locally. This is because the WMI call to Win32_Group.GetRelated() returns other stuff as well. So the question posed was “how do I get the list of Local Administrators locally.” More specifically, I want to monitor the local Administrators group.

I look at this two ways. Firstly, I want to get a regular list of names in the Administrators group and secondly, I want to monitor for changes to the …

» Continue reading

Quick PowerShell Script to Start Splunk

Got another quick PowerShell post for you. I have a copy of Splunk running locally on my Windows 8.1 workstation. I don’t always leave it running, for obvious resource reasons, therefor I end up starting it and stopping it as needed. On Windows, there’s two ways to control the Splunk services:

  • CLI splunk.exe start|stop|restart commands
  • Windows native service control methods (and there’s a half-dozen ways to do that)

So, in PowerShell, you can just do this:

Get-Service splunk* | Start-Service

The only minor problem is that I keep forgetting to elevate my PowerShell shell, so I’ll get an error message, and then I have to open a new window, and then repeat the process.  That’s no way to automate, I said to myself, so I made this quick …

» Continue reading

Quick Tip: Upload Logs to Splunk from Windows PowerShell

I had a folder full of log files I wanted to index real quick in my local instance of Splunk. They won’t persist, so the right thing to do is to use the “oneshot” command (documented here). This can be done in the web UI, but I like doing stuff at the command line. I opened up PowerShell (elevated, as my Splunk instance runs as system) and tried this:

splunk add oneshot *.log

And this was the output:

In handler 'oneshotinput': unable to open file: path='C:\Users\Hal\temp\*.log' error='The filename, directory name, or volume label syntax is incorrect.'

It didn’t work! Ok, so my assumption was that Splunk would parse the wildcard and have at it. But no big deal, this is quick to …

» Continue reading

Install Splunk with PowerShell (2014 Edition)

One of our avid twitter followers asked how to reliably install the Splunk Universal Forwarder on a Windows host with PowerShell last week. I’ve posted about all the intricacies involved before but improvements in open-source tools for PowerShell have made it a whole lot easier. You can take a look at the original article, but follow along here instead. We’re going to walk through what’s involved.

Installing as a Local SYSTEM user is easy. Here is the recipe:

Invoke-Command –ComputerName S1,S2,S3 –ScriptBlock { `
New-PSDrive S –Root \\SPLUNK\Files -PSProvider FileSystem; `
Start-Process S:\splunkforwarder-6.1.1-207789-x64-release.msi `
    –Wait -Verbose –ArgumentList (`
        “AGREETOLICENSE=`”Yes`””, `
        “/Liwem!”, “C:\splunkinstall.log” ) `

Let’s recap what you need to do to install a Splunk Universal …

» Continue reading

Active Directory Replication and Windows Server 2012 R2

If you have upgraded your Active Directory domain to Windows Server 2012 R2 and use the Splunk App for Active Directory, you may have noticed that the replication statistics script doesn’t work the same way as on older versions of Windows. Specifically, the ad-repl-stats.ps1 script takes forever to run and consumes just about as much memory as you can give it. This is because of a change in the implementation of the System.DirectoryServices.ActiveDirectory API that Microsoft provides. In prior releases of Windows Server, the API was lazy – data was only filled in within the objects when the data was requested. In Windows Server 2012 R2, those same objects filled in the data at instantiation. When we read the …

» Continue reading

Logging DMVs from Microsoft SQL Server with PowerShell

Some systems are easy to monitor and diagnose – just Splunk the log file or performance counter and you are pretty much done. Others take a little more work. Take, for example, Microsoft SQL Server. Many of the best bits of management information are stored in Dynamic Management Views, or DMVs. Getting to them is not so straight forward.

In order to get those nuggets, we need to do some pre-work. Firstly, install a Splunk Universal Forwarder on the SQL Server. Then fire up the SQL Server Management Studio and add the LOCAL SYSTEM account to the sysadmin role. This will allow the local machine access to all the information you need to monitor any database within the instances …

» Continue reading

Splunking Windows PowerShell Commands

This years user conference was another great conference and we got a ton of questions from you during the conference. Some of them I couldn’t answer at the time – I’m making up for that in between blog posts about new features. The first one was “Is there any way I can splunk what PowerShell commands are being executed on a server?”

There are two pieces of this puzzle: firstly – can I turn on an audit log that includes all the PowerShell commands that are executed within the system? We do that normally through group policy. Open up the group policy management console and take yourself to:

Computer Configuration\Administrative Templates\Windows Components\Windows PowerShell

In this group policy container there is …

» Continue reading

Monitoring Windows File Share Permissions with Splunk and PowerShell

I stopped my last blog post on Windows File Shares noting that there was still more to do. Monitoring Windows File Shares is a three part puzzle:

  1. Accesses
  2. Share Changes
  3. Permission Changes

We have already handled the first two, so this blog post is all about the final one – monitoring permission changes.

Let’s first consider how one would do this generically. As with the file shares, there is a WMI class for monitoring permissions, but it’s harder to use. You need to do it on a per-share basis, like this:

gwmi Win32_LogicalShareSecuritySetting -Filter "Name='$shareName'"

The Win32_LogicalShareSecuritySetting is a complex beast. Fortunately, we only need to know a couple of things. The most important one is the security descriptor. You …

» Continue reading

Monitoring Windows Shares with Splunk and PowerShell

I sometimes get emails after blog posts. One of the (fair) criticisms is that I sometimes do something in PowerShell that can be quite legitimately done via another data input like WMI. While this is true for simple cases, it’s not always true. Take the request, for example, of monitoring network shares. There are three parts to this. Firstly, producing a monitor of the share itself; secondly, producing a monitor of the permissions on the share; and finally, monitoring the file accesses utilizing that share. I’ve already blogged about the last one. Let’s take a look at the first two.

You can actually monitor the share itself using WMI. Network Shares are exposed via a WMI class Win32_Share. …

» Continue reading