Getwatchlist: getting watchlists into Splunk quickly and easily with a Splunk custom search command

As a Splunk partner specializing in Federal deployments, one question Aplura consultants are repeatedly asked by our clients is “Can I use Splunk to check our events for matches against a watchlist of IP addresses or domain names?”. Of course, the answer is “yes” watchlists can be configured, leveraging Splunk’s “lookup” functionality, and then used in searches to find and alert on matches. Splunk makes this pretty easy. Find more on lookups in the Splunk docs.

Implement Watchlists as Lookups in Splunk
For many of our clients, managing the lookup tables can be challenging The watchlist gets downloaded, perhaps reformatted, then uploaded to the Splunk server, where it can be used as a lookup table. In some situations, however, administrators aren’t comfortable giving their users direct access to upload files to the Splunk server and make the associated configuration changes. They are looking for another way to allow their users to save and update watchlists in Splunk, and preferably, keep those watchlists up to date in an automated fashion.

Extend Splunk Search with Custom Search Commands
After discussing the problem with a colleague, we came up with an idea. One of the great aspects of Splunk is its flexibility and extendability, which lead us to custom search commands. Custom search commands allow you to extend Splunk searches using the Python scripting language. Our solution to getting watchlists into Splunk is a custom search command called “getwatchlist”.

Getwatchlist allow Splunk users to retrieve delimited files from HTTP, HTTPS or FTP sources and then transform them into CSV output, which can easily be ingested by Splunk and saved as lookup tables. The lookup tables can then be used in searches to comb through events for hits. Usage is simple from the search bar or, for added convenience, through stored profiles in a configuration file. As getwatchlist is just using a Splunk search to populate the lookup table, scheduled searches can be configured to update the lists regularly, without requiring the types of access to the Splunk server that might make system administrators nervous.

Watchlists In Action (Using Splunk)
Note: Before proceeding with any of these examples, download and install the getwatchlist app from Splunkbase.

For example, let’s say an organization has a problem with users falling for phishing emails and clicking on the embedded links. Using Splunk and their web proxy events, we can use the PhishTank list to compare the URLs in our web proxy event to known phishing URLs.

First we need to pull down the latest copy of the PhishTank list using getwatchlist:

| getwatchlist http://data.phishtank.com/data/online-valid.csv delimiter=”,” relevantFieldName=url relevantFieldCol=2 referenceCol=3 dateCol=4 categoryCol=8 ignoreFirstLine=true isbad=true | outputlookup phishtank.csv

Now we compare the URLs in the lookup to the field in our events:

eventtype=webproxy [| inputlookup phishtank.csv | fields url]

If our administrator has been nice enough to configure phishtank as a lookup, we can also use this syntax to do the same as above, more simply:

eventtype=webproxy | lookup phishtank url | search isbad=true

The “isbad=true” at the end is a field and value which returns true for every column in the lookup (this is a custom field added by the getwatchlist command above).

So what about IP addresses? For example, let’s check our firewall logs for connections to known Command and Control servers using the AmaDa Malware Database.

Get (or update) the watchlist:

| getwatchlist http://amada.abuse.ch/blocklist.php?download=ipblocklist delimiter=# categoryCol=2 isbad=true | outputlookup amada.csv

And then check out events, matching the ip_address from the watchlist to the dest_ip field in the firewall events:

sourcetype=firewall [|inputlookup amada.csv | fields ip_address | rename ip_address as dest_ip]
or
sourcetype=firewall | lookup amada ip_address as dest_ip | search isbad=true

Another use would be for internal lists. If we have a list of terminated users, we can check our events to make sure these users aren’t logging in.

| getwatchlist http://192.168.1.104/bad_users.csv relevantFieldName=user isbad=true | outputlookup terminated_users.csv

The check our login events to make sure we don’t see any login activity:

tag=login |[inputlookup terminated_users.csv | fields user]
or
tag=login | lookup terminated_users user | search isbad=true

We were lucky enough to debut getwatchlist at the SplunkLiveDC! event hosted by Splunk Federal recently in Washington, DC. We were thrilled to see how excited people were about this, as well as their Splunk enthusiasm. See it again this week at #splunkconf.

Get Getwatchlist

Thanks to David Shpritz from Aplura, another awesome Splunkbase and Splunk Answers contributor, plus Federal systems integrator partner extraordinaire!