Monitoring Network Traffic with Sysmon and Splunk

Every IT guy has a set of tools that they use every day. One of mine is sysinternals. It’s a set of Windows utilities made available by Microsoft that do a whole slew of things. You can install them with chocolatey (another in my toolset) or downloaded and unpacked from their website. If you use Windows and this toolset isn’t in your arsenal, maybe it’s time.

Back in August, I got a request from one of our engineers asking me if we had any plans to support the collection of Sysmon data. Sysmon is a Windows system service (yes, another agent) that logs system activity to the Windows Event Log. However, it places all the important stuff in the XML data block – that bit of the Windows Event Log that we did not expose until 6.2.0. Now that we have the renderXml parameter on WinEventLog, we can do something about it. I was reminded of this utility last week when one of our security researchers asked about network connections.

Installing it is relatively simple. Download the package, unzip it, then run:

sysmon -i -n -accepteula

The -i installs the service and the -n instructs it to monitor network connections. Once that is done, the service will start dropping events in the stream Application and Services Logs/Microsoft/Windows/Sysmon/Operational. You can receive these logs in Splunk by using the following inputs.conf entry:

[WinEventLog://Microsoft-Windows-Sysmon/Operational]
disabled = false
renderXml = true

Now that you have events in Splunk, there is a wealth of information available to you. The basic search is:

sourcetype="XmlWinEventLog:Microsoft-Windows-Sysmon/Operational"

The normal EventCode field is available and there are two basic event codes available to you. There are several event codes generated, but the two you will want to be concerned with are EventCode=”1″, which is a process creation and EventCode=”3″, which is a network connection.

Which brings us back to our problem. Last week, one of our security researchers was asking about monitoring network connections. Most specifically, he wanted outbound connections, the process ID of the process that created the outbound connection and the parent process ID of that process. As he explained it, if you see a connection to a known bad IP or domain and you suspect malware, then the process ID of the process opening the connection is the malware and the parent process is likely the malware dropper. We have all this information, so let’s take a look.

First of all, we have our outbound connection. The events look like this:

<Event xmlns='http://schemas.microsoft.com/win/2004/08/events/event'><System><Provider Name='Microsoft-Windows-Sysmon' Guid='{5770385F-C22A-43E0-BF4C-06F5698FFBD9}'/><EventID>3</EventID><Version>3</Version><Level>4</Level><Task>1</Task><Opcode>0</Opcode><Keywords>0x8000000000000000</Keywords><TimeCreated SystemTime='2014-11-24T18:54:04.495825000Z'/><EventRecordID>61155</EventRecordID><Correlation/><Execution ProcessID='3360' ThreadID='2796'/><Channel>Microsoft-Windows-Sysmon/Operational</Channel><Computer>SPLUNK.ahall.local</Computer><Security UserID='S-1-5-18'/></System><EventData><Data Name='UtcTime'>11/24/2014 6:54 PM</Data><Data Name='ProcessGuid'>{00000451-4DA3-5459-0000-0010E8F30000}</Data><Data Name='ProcessId'>884</Data><Data Name='Image'>C:\Windows\system32\svchost.exe</Data><Data Name='User'>NT AUTHORITY\NETWORK SERVICE</Data><Data Name='Protocol'>udp</Data><Data Name='Initiated'>false</Data><Data Name='SourceIsIpv6'>true</Data><Data Name='SourceIp'>ff02:0:0:0:0:0:1:3</Data><Data Name='SourceHostname'></Data><Data Name='SourcePort'>5355</Data><Data Name='SourcePortName'>llmnr</Data><Data Name='DestinationIsIpv6'>true</Data><Data Name='DestinationIp'>fe80:0:0:0:354c:3220:8159:7591</Data><Data Name='DestinationHostname'>SPAPP10</Data><Data Name='DestinationPort'>50739</Data><Data Name='DestinationPortName'></Data></EventData></Event>

In order to get this properly extracted, we need to do some work with props and transforms. Fortunately, Splunk provides a KV_MODE of xml that extracts some of the data. However the Data elements need to be extracted separately and some of the automated extractions didn’t work, so I rolled my own. Here is my props.conf:

[XmlWinEventLog:Microsoft-Windows-Sysmon/Operational]
REPORT-sysmon = sysmon-eventid,sysmon-version,sysmon-level,sysmon-task,sysmon-opcode,sysmon-keywords,sysmon-created,sysmon-record,sysmon-correlation,sysmon-channel,sysmon-computer,sysmon-sid,sysmon-data

And here is my transforms.conf:

[sysmon-eventid]
REGEX = <EventID>(\d+)</EventID>
FORMAT = EventCode::$1

[sysmon-version]
REGEX = <Version>(\d+)</Version>
FORMAT = Version::$1

[sysmon-level]
REGEX = <Level>(\d+)</Level>
FORMAT = Level::$1

[sysmon-task]
REGEX = <Task>(\d+)</Task>
FORMAT = Task::$1

[sysmon-opcode]
REGEX = <Opcode>(\d+)</Opcode>
FORMAT = Opcode::$1

[sysmon-keywords]
REGEX = <Keywords>(0x[0-9a-fA-F]+)</Keywords>
FORMAT = Keywords::$1

[sysmon-created]
REGEX = <TimeCreated SystemTime='(.*?)'/>
FORMAT = TimeCreated::$1

[sysmon-record]
REGEX = <EventRecordID>(\d+)</EventRecordID>
FORMAT = RecordID::$1

[sysmon-correlation]
REGEX = <Correlation>(.*?)</Correlation>
FORMAT = Correlation::$1

[sysmon-channel]
REGEX = <Channel>(.*?)</Channel>
FORMAT = EventChannel::$1

[sysmon-computer]
REGEX = <Computer>(.*?)</Computer>
FORMAT = Computer::$1

[sysmon-sid]
REGEX = <Security UserID='(S-[0-9a-fA-f-]+)'/>
FORMAT = SecurityID::$1

[sysmon-data]
REGEX = <Data Name='(.*?)'>(.*?)</Data>
FORMAT = $1::$2

With full extraction, we can now do some interesting searches. Here is a search that shows all the outbound connections that a particular process uses (together with the user and computer that is running it):

sourcetype="XmlWinEventLog:Microsoft-Windows-Sysmon/Operational" EventCode=3 Protocol=tcp Initiated=true | eval src=if(isnotnull(SourceHostname), SourceHostname+":"+SourcePort, SourceIp+":"+SourcePort)  | eval dest=if(isnotnull(DestinationHostname), DestinationHostname+":"+DestinationPort, DestinationIp+":"+DestinationPort)  | eval src_dest=src + " => " + dest | stats values(src_dest) as Connection by ProcessGuid ProcessId User Computer Image

Similarly, you can lookup processes as they are created:

sourcetype="XmlWinEventLog:Microsoft-Windows-Sysmon/Operational" EventCode=1 NOT User="NT AUTHORITY\\SYSTEM" | stats values(User) as User,values(CommandLine) as CommandLine,values(ProcessId) as ProcessId,values(ParentProcessId) as ParentProcessId values(ParentCommandLine) as ParentCommandLine by LogonGuid

Note that we are exempting NT AUTHORITY\SYSTEM (and don’t forget the double-backslash) because we literally get thousands of them.

How about getting the command line that was run for each outbound non-local network connection?

sourcetype="xmlwineventlog:microsoft-windows-sysmon/operational" EventCode=3 Protocol=tcp Initiated=true | where DestinationIp!="127.0.0.1" AND DestinationHostname!=SourceHostname | table _time User Computer ProcessId ProcessGuid DestinationHostname DestinationPort | join type=inner [search sourcetype="xmlwineventlog:microsoft-windows-sysmon/operational" EventCode=1 | table _time ProcessGuid ProcessId CommandLine]

I’m currently working on releasing a CIM-compliant TA-microsoft_sysmon to handle these use cases. You can download the latest version from my GitHub repository, or watch for it arriving on apps.splunk.com soon.

I was looking for this. #lifesaver. Thanks Adrian!

November 25, 2014

sweeeeeeeeet! :)

lmyrefelt
November 27, 2014

Awesome! This helps us a lot! Well done, Adrian!

Martin
November 27, 2014

Brings us ever-closer to approximating true ETDR capabilities by simply using the Universal Forwarder and semi-native Windows features. Well done.

December 4, 2014

Just getting this all set up now.. -Thanks!

The next logical step with sysmon sourcetype is blacklisitng of noisy events having process information you don’t want to forward. With this wineventlog based sourcetype being rendered as Xml, it seems as if advanced blacklist filters do not work.. I’ve been trying to get advanced filtering to work with _raw key but that does not seem to be supported. Are there plans to support blacklisting with regex on _raw key?

David
March 22, 2015

Just as an FYI, the new version of Sysmon (2.0) has an option called ‘image filter’ that allows you to exclude events in a config xml, however there is a bug in the current version that is preventing the filtering from happening. I’ve been told that this will be corrected in the next version in a few weeks, and in addition they will be simplifying the configuration file to be easier to understand and review.

Jeff
April 8, 2015

Can’t find the splunk add-on. Is anyone still using this? I am very interested in it.

Todd
May 29, 2015

@David – you can always set your forwarder to not render this data in XML (instead standard KV pairs) and then regex should work better. @Jeff I believe we have chatted – in 3.1 of Sysmon you can do filtering. I will be sharing some findings at .conf during my “Splunking the Endpoint” talk.

September 9, 2015

@Todd the add-on is on Splunkbase – simply go there and search “sysmon.” @Jeff I believe we have chatted, and in 3.1 the filtering works. @David you can always set the output of this sourcetype to NOT “renderXml” and then you could regex match whatever you wanted. I will be sharing some Sysmon config details, along with observed volumes, at .conf in my “Splunking The Endpoint” talk.

James Brodsky
September 9, 2015