Correlating with Splunk IP Watchlist

Last year, fellow Splunker, Dave Croteau, created a prototype to daily index the world’s top 100 suspicious, or in some cases malicious IP addresses, by using a list created by the dshield.org web site. One thought is that these addresses may be compromised by trojans or botnets, so you would not want them to appear as sources connecting to your network. Dave also used the Splunk Maxmind add-on to show a simple dashboard that map’s these addresses to country and city with the Splunk Top command.

Country and City of Offending IP Addresses

Next, I took this app and changed the scripted input to use curl to gather the data so the same approach could be ported to Windows as well as *Nix based machines. I also added a view to show this data on an AmMap dashboard.

Map of locations for offending IP addresses

You can download this complete app from Splunkbase. For reasons of testing, I purposely put the events in the included sample index, but you are free to change the inputs.conf file to use any index you like. For now, you may say that these are just pretty pictures to view from Splunk for where possible malicious IP addresses are coming from on a daily basis. That’s not the goal of this app. What should be done is to run a scheduled saved search on a regular basis to correlate the IP addresses from this app to any of the source (or even destination addresses) addresses that are in your own log files. If there is a match, then an alert should be executed. For example, if your in bound Apache access logs have a source IP address of say, xxx.23.34.12 (I’ve masked out the first octet as not to accidentally put in some site’s real IP address) and this is considered a possible malicious IP address from the Dshield web site, then you would want to know about.

One way to do this in Splunk is to first use Splunk Web’s Manager to create an alias field for your access log’s source IP address. Let’s call it questionableIP. Next, create the same alias, questionableIP, for the offending_ip field for the ip_watchlist sourcetype. Your scheduled saved search running every day could then look like:

(index=sample OR index=main) (sourcetype=access_common OR sourcetype=ip_watchlist)|transaction questionableIP maxspan=1d|eval stcount=mvcount(sourcetype)|where stcount=2

What this search says is group all events from either IPWatchList and the access_common sourcetypes where the questionableIP field is in common. Since this is using OR for the sourcetypes, the last part of the search says count the number of sourcetypes and only report results if the number of sourcetypes is equal to two. For your alert conditions, if the number of results returned is greater than 0, you will want to know about this.

This is just one way to correlate with this daily indexed data. You may also consider using other Splunk techniques such as subsearches. Once you find a malicious IP address in your events, you may also want to go back in time a little further to see if this IP address was also in your logs in the past in case you want to investigate active persistent threats. I hope this makes this app a little more useful.