Splunk 6.2 Feature Overview: XML Event Logs

We’ve been (rightly) criticized for a couple of things in recent years. Firstly, when you configure a Windows Event Log, it’s too big. This is because we combine the event log object with the message from the locale-specific DLL and that includes a bunch of common explanatory text. I don’t really need to know what a login really means (to the tune of 1K of data ingest) every time someone logs in, especially when these events are happening hundreds of times a minute. Secondly, our event log extractions are for US/English only. Got German Windows? Sorry – our extractions don’t work for that. Finally, we discard the additional data that is provided in the event log object. A primary example of this is the Logon Performance Report, which Jason Conger has written about. In addition, we occasionally lost contact with the locale-specific message DLL and that caused us to not write the event at all, losing even more data.

The locale-specific message DLL is the culprit here. It changes depending on the locale of Windows you have installed, causes the message bloat and throws away data that we are interested in. What if we didn’t use the locale-specific message DLL? What if we stored the event logs in XML?

With Splunk 6.2, that’s exactly what you can do. The effort does not come for free, however. We’ve already done the hard work of updating the Splunk_TA_Windows to include CIM compliant extractions for the Security Event Log. If you have done your own extractions (or you are using an alternate TA that contains extractions), then these need to be updated to support the XML format. However, converting to the XML format solves all the problems I mentioned.

So, how does it work? First of all, you need to update your inputs.conf to render the XML. We’ve designed it so that nothing changes on upgrade (backwards compatibility FTW!) You need to add a line to your WinEventLog definitions like this:

[WinEventLog://Security]
index=security
current_only=1
evt_resolve_ad_obj=0
renderXml=1
disabled=0

Once you have restarted your universal forwarder, you will see XML coming in with a different source type (XmlWinEventLog:Security, for example).

<Event xmlns='http://schemas.microsoft.com/win/2004/08/events/event'>
 <System>
  <Provider Name='Microsoft-Windows-Security-Auditing' Guid='{54849625-5478-4994-A5BA-3E3B0328C30D}'/>
  <EventID>4672</EventID>
  <Version>0</Version>
  <Level>0</Level>
  <Task>12548</Task>
  <Opcode>0</Opcode>
  <Keywords>0x8020000000000000</Keywords>
  <TimeCreated SystemTime='2014-04-29T22:15:03.280843700Z'/>
  <EventRecordID>2756</EventRecordID>
  <Correlation/>
  <Execution ProcessID='540' ThreadID='372'/>
  <Channel>Security</Channel>
  <Computer>sacreblue</Computer>
  <Security/>
 </System>
 <EventData>
  <Data Name='SubjectUserSid'>AUTORITE NT\Système</Data>
  <Data Name='SubjectUserName'>Système</Data>
  <Data Name='SubjectDomainName'>AUTORITE NT</Data>
  <Data Name='SubjectLogonId'>0x3e7</Data>
  <Data Name='PrivilegeList'>SeAssignPrimaryTokenPrivilege
                             SeTcbPrivilege
                             SeSecurityPrivilege
                             SeTakeOwnershipPrivilege
                             SeLoadDriverPrivilege
                             SeBackupPrivilege
                             SeRestorePrivilege
                             SeDebugPrivilege
                             SeAuditPrivilege
                             SeSystemEnvironmentPrivilege
                             SeImpersonatePrivilege</Data>
 </EventData>
</Event>

Note that the keys are always rendered in English, irrespective of the system locale of the machine that is generating the events. This event was generated from a machine running with French as the system locale – note the field names.

The effect of this on your data ingest rate is rather dramatic. Firstly, the size of the Windows Security Event Log is reduced by almost 70%. That means that a typical Active Directory Monitoring solution will occupy just over 1Gb of data per 1,000 users (instead of the 4Gb of data that it would take up in the old format). In addition, you don’t have to write custom code to get at those extra fields – the Logon timing report discussed above is now possible with just a simple dashboard.

For those of you running Enterprise Security, you will be happy to hear that, as of Splunk_TA_Windows 4.7.0, the XmlWinEventLog format is recognized within the TA and all the same field extractions are performed on the new format. That means you can take advantage of the data ingest savings straight away.

You might be wondering if there is any reason to not use the XmlWinEventLog format as your default now? Unfortunately, yes, there is. When we wrote the Windows Infrastructure and Exchange apps, we used the old format. The field extractions for these apps will be updated in the next release. Some dashboards will not render correctly if you change your data format right now.

Adrian,

Does a config change need to be made on the domain controllers to store the event logs in XML format?

New to the Splunk_TA_Windows 4.7.0 and trying to get up to speed.

Thx

Jeff
November 5, 2014

Yes – you will need to change the inputs.conf to enable the new format.

November 5, 2014

Hello Adrian,

seems to be a bug with renderXML, according btool it has to be renderXml (ml lowercase!).
Can you look into it? with uppercase XML it produces an error!

Kind regards

noOne
November 11, 2014

Me again, is it possible that message= is missing in the xml structure?

noOne
November 11, 2014

Hi Adrian,
I have my own agent to capture windows event logs and store them on XML format on a directory. I see some documentation on how to monitor directories to index data, but I don’t see any specifics on how to do it for XML windows events. Can you help?

Gustavo Armenta
December 15, 2014

If they are .evtx files, you can just monitor:// the the directory in inputs.conf. However, they will be decoded in the normal manner to be non-XML via the Messaging DLL. If you want them in XML, then rename the files to .xml and use props/transforms to ensure that the data is appropriately segmented.

December 15, 2014

I’ve been receiving a fair bit of email regarding this blog post and the purported savings. Our test of approx. 70% saving was with the Security Windows Event Log as it is configured in the Splunk_TA_winodws and with all audit turned on with a Windows 2012R2 domain controller. Each security event has a block of text at the end of the event which describes what the event is. This text is recorded with each event. You can do other things to suppress the message.

With other event log channels (without explanatory text), the sizes are about the same. With message suppression turned on, the Xml event logs will be approximately 10-15% bigger.

December 17, 2014

What version do the universal forwarders have to be on to support this?

Troy
December 22, 2014

XML event logs are supported on Splunk v6.2.0 and later.

December 24, 2014

I have tried XML event logs with Enterprise Security. Unfortunately the charts are totaly bit off, because ES is expecting fields action that comes from field “type”. Is there any modification for ES or Windows TA to cover this issue?

February 23, 2015

Some dashboard in window infrastructure app might be cannot display when we convert to XML. Because we cut some information from log. Is it possible to make sure Windows Infra Dashboard still work correctly?

Thu Ho
July 30, 2015

We discovered (after significant investment) that the “renderXml” option does not work for Windows Server 2003 (and others?). What’s so tragic is that the documentation does not mention tihs so a note should be added (similar to the notes on other options that mention OS in/compatability) to the “renderXml” option to warn others so they do not suffer the same wasted effort that we did in an option that does not work for all servers.

Gregg Woodcock
October 2, 2015

One Trackback

  1. […] week, I covered the XML Event Logs – an awesome feature that will reduce your data ingest, increase the fidelity of the data […]