Monitoring Network Traffic with Sysmon and Splunk

Every IT guy has a set of tools that they use every day. One of mine is sysinternals. It’s a set of Windows utilities made available by Microsoft that do a whole slew of things. You can install them with chocolatey (another in my toolset) or downloaded and unpacked from their website. If you use Windows and this toolset isn’t in your arsenal, maybe it’s time.

Back in August, I got a request from one of our engineers asking me if we had any plans to support the collection of Sysmon data. Sysmon is a Windows system service (yes, another agent) that logs system activity to the Windows Event Log. However, it places all the important stuff in the XML data block – that bit of the Windows Event Log that we did not expose until 6.2.0. Now that we have the renderXml parameter on WinEventLog, we can do something about it. I was reminded of this utility last week when one of our security researchers asked about network connections.

Installing it is relatively simple. Download the package, unzip it, then run:

sysmon -i -n -accepteula

The -i installs the service and the -n instructs it to monitor network connections. Once that is done, the service will start dropping events in the stream Application and Services Logs/Microsoft/Windows/Sysmon/Operational. You can receive these logs in Splunk by using the following inputs.conf entry:

[WinEventLog://Microsoft-Windows-Sysmon/Operational]
disabled = false
renderXml = true

Now that you have events in Splunk, there is a wealth of information available to you. The basic search is:

sourcetype="XmlWinEventLog:Microsoft-Windows-Sysmon/Operational"

The normal EventCode field is available and there are two basic event codes available to you. There are several event codes generated, but the two you will want to be concerned with are EventCode=”1″, which is a process creation and EventCode=”3″, which is a network connection.

Which brings us back to our problem. Last week, one of our security researchers was asking about monitoring network connections. Most specifically, he wanted outbound connections, the process ID of the process that created the outbound connection and the parent process ID of that process. As he explained it, if you see a connection to a known bad IP or domain and you suspect malware, then the process ID of the process opening the connection is the malware and the parent process is likely the malware dropper. We have all this information, so let’s take a look.

First of all, we have our outbound connection. The events look like this:

<Event xmlns='http://schemas.microsoft.com/win/2004/08/events/event'><System><Provider Name='Microsoft-Windows-Sysmon' Guid='{5770385F-C22A-43E0-BF4C-06F5698FFBD9}'/><EventID>3</EventID><Version>3</Version><Level>4</Level><Task>1</Task><Opcode>0</Opcode><Keywords>0x8000000000000000</Keywords><TimeCreated SystemTime='2014-11-24T18:54:04.495825000Z'/><EventRecordID>61155</EventRecordID><Correlation/><Execution ProcessID='3360' ThreadID='2796'/><Channel>Microsoft-Windows-Sysmon/Operational</Channel><Computer>SPLUNK.ahall.local</Computer><Security UserID='S-1-5-18'/></System><EventData><Data Name='UtcTime'>11/24/2014 6:54 PM</Data><Data Name='ProcessGuid'>{00000451-4DA3-5459-0000-0010E8F30000}</Data><Data Name='ProcessId'>884</Data><Data Name='Image'>C:\Windows\system32\svchost.exe</Data><Data Name='User'>NT AUTHORITY\NETWORK SERVICE</Data><Data Name='Protocol'>udp</Data><Data Name='Initiated'>false</Data><Data Name='SourceIsIpv6'>true</Data><Data Name='SourceIp'>ff02:0:0:0:0:0:1:3</Data><Data Name='SourceHostname'></Data><Data Name='SourcePort'>5355</Data><Data Name='SourcePortName'>llmnr</Data><Data Name='DestinationIsIpv6'>true</Data><Data Name='DestinationIp'>fe80:0:0:0:354c:3220:8159:7591</Data><Data Name='DestinationHostname'>SPAPP10</Data><Data Name='DestinationPort'>50739</Data><Data Name='DestinationPortName'></Data></EventData></Event>

In order to get this properly extracted, we need to do some work with props and transforms. Fortunately, Splunk provides a KV_MODE of xml that extracts some of the data. However the Data elements need to be extracted separately and some of the automated extractions didn’t work, so I rolled my own. Here is my props.conf:

[XmlWinEventLog:Microsoft-Windows-Sysmon/Operational]
REPORT-sysmon = sysmon-eventid,sysmon-version,sysmon-level,sysmon-task,sysmon-opcode,sysmon-keywords,sysmon-created,sysmon-record,sysmon-correlation,sysmon-channel,sysmon-computer,sysmon-sid,sysmon-data

And here is my transforms.conf:

[sysmon-eventid]
REGEX = <EventID>(\d+)</EventID>
FORMAT = EventCode::$1

[sysmon-version]
REGEX = <Version>(\d+)</Version>
FORMAT = Version::$1

[sysmon-level]
REGEX = <Level>(\d+)</Level>
FORMAT = Level::$1

[sysmon-task]
REGEX = <Task>(\d+)</Task>
FORMAT = Task::$1

[sysmon-opcode]
REGEX = <Opcode>(\d+)</Opcode>
FORMAT = Opcode::$1

[sysmon-keywords]
REGEX = <Keywords>(0x[0-9a-fA-F]+)</Keywords>
FORMAT = Keywords::$1

[sysmon-created]
REGEX = <TimeCreated SystemTime='(.*?)'/>
FORMAT = TimeCreated::$1

[sysmon-record]
REGEX = <EventRecordID>(\d+)</EventRecordID>
FORMAT = RecordID::$1

[sysmon-correlation]
REGEX = <Correlation>(.*?)</Correlation>
FORMAT = Correlation::$1

[sysmon-channel]
REGEX = <Channel>(.*?)</Channel>
FORMAT = EventChannel::$1

[sysmon-computer]
REGEX = <Computer>(.*?)</Computer>
FORMAT = Computer::$1

[sysmon-sid]
REGEX = <Security UserID='(S-[0-9a-fA-f-]+)'/>
FORMAT = SecurityID::$1

[sysmon-data]
REGEX = <Data Name='(.*?)'>(.*?)</Data>
FORMAT = $1::$2

With full extraction, we can now do some interesting searches. Here is a search that shows all the outbound connections that a particular process uses (together with the user and computer that is running it):

sourcetype="XmlWinEventLog:Microsoft-Windows-Sysmon/Operational" EventCode=3 Protocol=tcp Initiated=true | eval src=if(isnotnull(SourceHostname), SourceHostname+":"+SourcePort, SourceIp+":"+SourcePort)  | eval dest=if(isnotnull(DestinationHostname), DestinationHostname+":"+DestinationPort, DestinationIp+":"+DestinationPort)  | eval src_dest=src + " => " + dest | stats values(src_dest) as Connection by ProcessGuid ProcessId User Computer Image

Similarly, you can lookup processes as they are created:

sourcetype="XmlWinEventLog:Microsoft-Windows-Sysmon/Operational" EventCode=1 NOT User="NT AUTHORITY\\SYSTEM" | stats values(User) as User,values(CommandLine) as CommandLine,values(ProcessId) as ProcessId,values(ParentProcessId) as ParentProcessId values(ParentCommandLine) as ParentCommandLine by LogonGuid

Note that we are exempting NT AUTHORITY\SYSTEM (and don’t forget the double-backslash) because we literally get thousands of them.

How about getting the command line that was run for each outbound non-local network connection?

sourcetype="xmlwineventlog:microsoft-windows-sysmon/operational" EventCode=3 Protocol=tcp Initiated=true | where DestinationIp!="127.0.0.1" AND DestinationHostname!=SourceHostname | table _time User Computer ProcessId ProcessGuid DestinationHostname DestinationPort | join type=inner [search sourcetype="xmlwineventlog:microsoft-windows-sysmon/operational" EventCode=1 | table _time ProcessGuid ProcessId CommandLine]

I’m currently working on releasing a CIM-compliant TA-microsoft_sysmon to handle these use cases. You can download the latest version from my GitHub repository, or watch for it arriving on apps.splunk.com soon.