November Spawned an Osquery

This blog post is part twenty-one of the "Hunting with Splunk: The Basics" series. This series is now legally allowed to drink! (Which is relevent when dealing with James Brodsky as a contributing author.) osquery has quickly become one of the most used resources for gathering data on *nix and MacOSs. However, along with its ability for real-time information pulls, it's also a phenomonal tool for logging. With that in mind, James goes over some interesting things he has found and how to do them! – Ryan Kovar

Author's Note: Ryan Kovar is a taskmaster. He doesn’t understand that my current job at Splunk is to “manage” a bunch of security people—not to actively do hands-on experimentation and write up explanations of security-useful data sources. However, he asked me to come out of semi-retirement for this blog on osquery. If you see him, tell him he owes me a chilled bottle of Loire Valley rosé and that I want my Smiths mixtapes back.

As part of recent work within the Security Specialist team at Splunk, we’ve been taking a closer look at sources of endpoint data that we're seeing across our customer base. It isn’t easy—every time I open my Twitter feed, someone's going off about the latest open-source this and next-gen that. Because guess what? It turns out you can throw all the endpoint++, machine learning, AI-based detection and prevention technology you want at your systems, but unless you’ve got 100% coverage, you still have risk. I base this on findings from the SANS 2018 Survey on Endpoint Protection. The overwhelming majority of people don’t fully implement what they buy.

(This is exactly what happened when I bought that P90x DVD last year so that I could look just like Doug Merritt.)

But I digress. One of the more prevalent endpoint data sources out there is osquery; let's go deeper.

Osquery is a fully open-sourced, scalable endpoint visibility project contributed to and sponsored by Facebook. Since it's multi-platform (Linux, Windows, and macOS) and tested at scale, it's increasingly likely to be found on corporate endpoints. Other companies such as Palantir and Kolide actively extend osquery, and the popularity of the project is such that QueryCon—an inaugural conference on all things osquery—was held in San Francisco this summer. (The videos from those sessions are excellent, high definition, sometimes irreverent, and can be found at that link as well.)

We’ve been seeing a gradual uptake in Splunk customers looking to analyze data at scale from osquery, especially on non-Windows endpoints. (We have yet to see a customer running osquery in production on Windows, and the one customer seriously considering it was such a huge Microsoft fan that they then gave us a VIP tour of their collection of Zunes.)

Customers looking for a solution for macOS endpoint monitoring, for example, often end up running osquery—as well as those looking to satisfy file integrity monitoring (FIM) requirements. The most common way to onboard osquery data is the time-tested Splunk Universal Forwarder, which can be configured to send local osquery output in real time directly into your Splunk indexers.

Why Should You Care?

Osquery is a fantastic tool for hunting through granular endpoint data. For example, in the Boss of the SOC (BOTS) version 2.0 dataset, we collected osquery data from two macOS endpoints. We used osquery for FIM capabilities against typical download destinations and configured customized process monitoring as well. This picked up a particularly poorly-written (but effective) piece of ransomware that we executed as part of the data generation for the competition; as you can see below, the full command line auditing was captured in Splunk. By using the data returned from “columns.cmdline,” competitors could actually decrypt files encrypted by the ransomware, peel back another layer of our warped sense of humor, and score lots of points! In fact, columns.cmdline (and really anything related to process monitoring with osquery) is some of the most useful security data harvested from Linux and macOS systems. (We’re still partial to Sysmon and 4688 Security events from Windows platforms).

sourcetype=”osquery_results” host=maclory-air13” name=”pack_osx_proclaunch_ProcessesInUserSpace”
| table _time,host,columns.cmdline,name,process,user
| dedup columns.cmdline
| reverse

Osquery is often configured with “query packs,” which are simply collections of osquery queries gathered together under a common purpose; much like the revolutionaries in Les Miserables. When you configure osquery to send data from standard query packs into Splunk, the “name” field is populated with the name of the pack that generated the data. Here, we’ve marked the data as coming from the two OS variants in our data generation environment (Windows and Linux.):

You’ll notice above that many of the query packs are focused on operational aspects of your systems. That’s right, osquery is just as useful for your IT Operations staff as it is for your security analysts. IT staff can quickly find performance issues and query their entire osquery fleet to check on configuration settings, installed apps, and hardware details, and then they can get back to sleep. But returning to security, you’ll notice many packs that include query categories like “Windows Hardening” and “Vulnerability Management” and “Incident Response” in the name.

What are examples of just some of the questions we can answer easily in Splunk with those packs?

• What Linux systems have had recent changes to /etc/hosts, and what are those changes?
• Which systems are suddenly listening on new ports (or no longer listening on existing ports), and what are those ports?
• Which users are currently logged in?
• What modifications have been made to Linux crontabs?
• Why does Arizona taunt me with their crazy timezone bullshit?
• Which Linux systems are now pointing to new APT repos?
• Which Windows systems are no longer configured to use UAC?
• Which Windows systems have potentially had a stickykeys backdoor installed?
• Which Linux kernel modules are currently in use?

Querying Splunk for this info couldn’t be simpler. When osquery finds a change of interest (based on the delta from the last time a check was run), it sends the data in JSON format directly into Splunk. Then, you simply query the pack name, hosts, and timeframe to see the output. Here’s an example of looking for that stickykey backdoor check from the standard Windows Attacks detection pack (this attack is when Windows binaries normally used for system accessibility are swapped out with binaries that can be used for evil, which is exactly what I did for this example):

index=botsv3 sourcetype="osquery:results"  name="pack_windows-attacks_StickyKeys_File_Replace_Backdoor"
| stats values(columns.path) as path, values(columns.sha256) as hash, values(decorations.username) as user by host

On October 1st, we invite you to come and play BOTS 3.0 at .conf18 and experience osquery data from Windows and Linux endpoints yourself! You’ll encounter the data from all of the packs listed above. You'll drink beer and eat expensive conference canapés. We’ll also be covering, in related BOTS sessions, some of the latest and greatest Splunk integration points with osquery, including a recently-updated app that turns your Splunk search head into a full-fledged osquery query server.

See you in Orlando!