Monitoring Network Traffic with Sysmon and Splunk

Every IT guy has a set of tools that they use every day. One of mine is sysinternals. It’s a set of Windows utilities made available by Microsoft that do a whole slew of things. You can install them with chocolatey (another in my toolset) or downloaded and unpacked from their website. If you use Windows and this toolset isn’t in your arsenal, maybe it’s time.

Back in August, I got a request from one of our engineers asking me if we had any plans to support the collection of Sysmon data. Sysmon is a Windows system service (yes, another agent) that logs system activity to the Windows Event Log. However, it places all the important stuff in the …

» Continue reading

Splunk App for SharePoint goes Open Source

For about the last year, I’ve been working on an update to the Splunk App for SharePoint. But it isn’t the one you would expect. I’ve been working to open source the app. At the end of the day the best person to write an IT Operations app for Splunk is the person who is intimately involved in the running of the workload. Today, we are flicking the switch and opening up the project. We are allowing you to directly file bugs and feature requests; we are allowing you to submit code; and we are encouraging you to get involved in the project.

So, how can you do this. Firstly, you will want to have some sort of test environment. …

» Continue reading

Splunk 6.2 Feature Overview: Perfmon Delocalization

Last week, I covered the XML Event Logs – an awesome feature that will reduce your data ingest, increase the fidelity of the data that is stored and allow us to work with localized data. Today, I want to discuss another localization feature – or at least a delocalization feature – perfmon.

Prior to Splunk 6.2, Windows perfmon was always collected localized. If you wanted the % Processor Time counter, you had to specify the localized version of this. If you were running on a french version of Windows, you would have to specify object=Processeur and counter=”% Temps Processeur” in both your inputs.conf and searches. Given that there are over 30 different localized versions of Windows, this really meant that …

» Continue reading

Splunk 6.2 Feature Overview: XML Event Logs

We’ve been (rightly) criticized for a couple of things in recent years. Firstly, when you configure a Windows Event Log, it’s too big. This is because we combine the event log object with the message from the locale-specific DLL and that includes a bunch of common explanatory text. I don’t really need to know what a login really means (to the tune of 1K of data ingest) every time someone logs in, especially when these events are happening hundreds of times a minute. Secondly, our event log extractions are for US/English only. Got German Windows? Sorry – our extractions don’t work for that. Finally, we discard the additional data that is provided in the event log object. A primary example …

» Continue reading

Integrating Active Directory into Splunk with SA-ldapsearch

On Tuesday, I introduced one of the first presentations at .conf2014 – a major update to the SA-ldapsearch app. This new app has now launched and you can download it at The app consists of four specific commands: ldapsearch, ldapfetch, ldapfilter and ldapgroup.

Improvements include:

  • We dropped the requirement for Java on your search head
  • We added support for Search Head Pooling
  • We added a GUI configuration page and connection testing
  • We provided full UTF-8 support

The ldapsearch command is a generating command and is used in a similar way to other generating commands like inputlookup. You run it like this:

| ldapsearch domain=SPL search="(objectClass=user)" attrs="sAMAccountName,cn"

We have added some new features in this release. Firstly, the output …

» Continue reading

What’s new in TA-windows 4.7.0?

If you are a Windows admin and use Splunk then you’ve likely deployed Splunk_TA_windows on your endpoints. It’s a central method for handling Windows data and has all the extractions you need to handle Windows event logs. We’ve just released version 4.7.0. So what’s new and should you upgrade?

The first thing we did was we organized the data. The well considered best practice is to not put data in the default index. Yet here we were putting data in the default index. That has now changed. By default, we create three indices for you:

  • perfmon is used for performance data
  • wineventlog is used for event logs
  • windows is used for everything else

This change will not affect you if …

» Continue reading

Monitoring Local Administrators on Windows Hosts

It is always gratifying when one of my readers comes to me with a problem. I love challenges. This one had to do with one of my old posts surrounding Local Administrators remotely. Of course, the way to do this is via WMI. However, it doesn’t quite work the same way locally. This is because the WMI call to Win32_Group.GetRelated() returns other stuff as well. So the question posed was “how do I get the list of Local Administrators locally.” More specifically, I want to monitor the local Administrators group.

I look at this two ways. Firstly, I want to get a regular list of names in the Administrators group and secondly, I want to monitor for changes to the …

» Continue reading

Install Splunk with PowerShell (2014 Edition)

One of our avid twitter followers asked how to reliably install the Splunk Universal Forwarder on a Windows host with PowerShell last week. I’ve posted about all the intricacies involved before but improvements in open-source tools for PowerShell have made it a whole lot easier. You can take a look at the original article, but follow along here instead. We’re going to walk through what’s involved.

Installing as a Local SYSTEM user is easy. Here is the recipe:

Invoke-Command –ComputerName S1,S2,S3 –ScriptBlock { `
New-PSDrive S –Root \\SPLUNK\Files -PSProvider FileSystem; `
Start-Process S:\splunkforwarder-6.1.1-207789-x64-release.msi `
    –Wait -Verbose –ArgumentList (`
        “AGREETOLICENSE=`”Yes`””, `
        “/Liwem!”, “C:\splunkinstall.log” ) `

Let’s recap what you need to do to install a Splunk Universal …

» Continue reading

Controlling 4662 Messages in the Windows Security Event Log

You’ve just installed the Splunk App for Windows Infrastructure, or its friend the Splunk App for Exchange. You’ve followed all the instructions, placed the Universal Forwarders on the domain controllers, and configured everything according to the documentation. Now your license is blowing up because you are getting too many EventCode=4662 in the Windows Security Event Log. How did this happen?

Security EventCode 4662 is an abused event code. It is used for directory access, like this:

An operation was performed on an object. 
Subject : 
    Account Name: EXCH2013$ 
    Account Domain: SPL 
    Logon ID: 0x177E5B394
    Object Server: DS 
    Object Type: domainDNS 
    Object Name: DC=spl,DC=com 
    Handle ID: 0x0 
    Operation Type: Object Access 
    Accesses: Control …
» Continue reading

Fixing Scripted Inputs in Tiered Deployments

The Splunk App for Microsoft Exchange has a useful lookup named ad_username. It takes the various forms that you can logon to a domain as (like DOMAIN\user and and normalizes them. Further, it then takes all the user aliases and normalizes them so adrian.hall is the same as ahall and that is the same as adrian. It’s really useful when you are trying to deal with domain accounts from a support functionality – you don’t have to know how they logged in – only what their official username is.

AD_Username is a scripted input written in Python and lives in the bin directory of the application directory. It relies on two files that live in the local directory called …

» Continue reading