Monitoring Processes on Windows

We get a lot of questions here at the Splunk Microsoft Practice – not just on our apps (which are awesome starting points for common Microsoft workloads), but also how to do specific things in Windows. One of the things I recently got asked was “how do I get a top-10 type report of processes on a system and who is running them?” This should be fairly straight-forward. After all, Microsoft provides a perfmon object called “Process” – maybe I can just monitor that. Unfortunately, the owner is not available. Ditto with WMI. Once I’ve exhausted the built-in methods of getting information, I turn to my favorite tool – PowerShell.

There are two methods of getting the list of processes on a system. Get-Process is the de-facto standard for getting a process list from PowerShell, but I prefer the WMI approach – Get-WmiObject –class win32_process. The reason for the choice is that the objects that you get back have a bunch of useful methods on them, one of which is GetOwner() that retrieves the owner of the process – just what we are looking for. You can always get the list of things you can do by piping the command to Get-Member. For example:

Get-WmiObject -class win32_process | Get-Member

In order to get the owner information into the objects, we have to do a little work. Joel Bennett assisted with this small scriptlet:

Get-WmiObject –class win32_process | 
    Add-Member -MemberType ScriptProperty -PassThru -Name Username -Value { 
        $ud = $this.GetOwner();  
        $user=$ud.Domain+"\"+$ud.User;  
        if ($user -eq "\") { "SYSTEM" } else { $user } 
    }

Although I have split this over multiple lines for readability, you should type this all on the same line. What this does is add an “Owner” property to each object in the pipeline, and it gets the value by called GetOwner() on the object. There is a special case when the process does not have an owner, and in this case, we set the owner to “SYSTEM”.

You will notice an awful lot of properties being returned when you run this command. We will fix that when we start importing it into Splunk. Speaking of which, how do we do that? We turn to one of my favorite addons – SA-ModularInput-PowerShell. You can download it from Splunkbase. This addon persists a PowerShell scripting host for running scripts and gathering the results. Any objects that are output by our script are converted into key-value pairs and sent on to Splunk. You need to install the .NET 4.5 framework and WinRM 3.0 as well as the Splunk Universal Forwarder for Windows.

Since the SA-ModularInput-PowerShell addon does not define any scripts, you need to add your script to the inputs.conf of an app. Our script would appear like this:

[powershell://Processes]
script = Get-WmiObject -class win32_process | Add-Member -MemberType ScriptProperty -PassThru -Name Username -Value { $ud = $this.GetOwner();  $user=$ud.Domain+"\"+$ud.User;  if ($user -eq "\") { "SYSTEM" } else { $user } }|select ProcessId, Name, Username, Priority, ReadOperationCount, WriteOperationCount, CreationDate, Handle, VirtualSize, WorkingSetSize, UserModeTime, ThreadCount
schedule = 0,15,30,45 * * ? * *
source = PowerShell
sourcetype = PowerShell:Process

Our script is fairly evident, but we have added a Select to limit the properties that are sent on to Splunk. I’ve picked some interesting ones around memory usage, thread counts and IOPS. The schedule will be recognizable as a cron-style scheduler. The SA-ModularInput-PowerShell is based on Quartz.NET – a well known open-source scheduling system for the .NET framework.

Once the data is flowing into Splunk (check the splunkd.log file if it isn’t), we need a search that will get us the processes at any given time. Here is my search:

sourcetype=PowerShell:Process | 
    stats count as Polls,
        latest(Name) as Name,
        latest(Username) as Username,
        latest(Priority) as Priority,
        max(ReadOperationCount) as ReadOperationCount,
        max(WriteOperationCount) as WriteOperationCount,
        latest(Handle) as Handle,
        max(VirtualSize) as VirtualSize,
        latest(WorkingSetSize) as WorkingSetSize,
        latest(UserModeTime) as UserModeTime,
        max(ThreadCount) as ThreadCount by host,ProcessId,CreationDate

Again, run this all together on the same line – it’s just split up for readability. We need the CreationDate field because a ProcessId can be recycled on a given host. By utilizing the host, ProcessId and CreationDate, we get a unique key to identify each process. I normally place useful searches like this in a macro – either by editing my macros.conf file or in the Manager. I’ve named my macro “all-windows-processes”.

So, what about that top-ten processes. Well, it depends on how you measure the top ten. Here are some interesting searches using that macro:

Top 10 Processes run by users that have the largest virtual memory footprint

`all-windows-processes` | search Username!=”SYSTEM” | top VirtualSize

Top 10 Processes that have the largest amount of disk activity

`all-windows-processes` | eval DiskActivity = ReadOperationCount + WriteOperationCount | top DiskActivity

Top 10 Users that are running the most processes

`all-windows-processes` | stats count by Username,host | top count

Top 10 longest running user processes

`all-windows-processes` | search Username!=”SYSTEM” | top Polls

Hopefully, this gives you some ideas on what you can do to monitor processes on your Windows systems, and if you are wondering how to monitor something on your Windows systems, let us know at microsoft@splunk.com or use Ask an Expert – just look for my picture.

Hi Splunkers

I was trying to install the said apps but i got a error message please see below.

Unable to initialize modular input “powershell” defined inside the app “SA-ModularInput-PowerShell”: Introspecting scheme=powershell: script running failed (exited with code -2146232576).

Hoping for your urgent response

best Regards
Cris
Splunker

September 9, 2013

Post a Comment

Your email is never published nor shared. Required fields are marked *

*
*