Security

July 21, 2024

4 Minute Read

Splunk Security Content for Impact Assessment of CrowdStrike Windows Outage

On July 19, 2024, CrowdStrike, a global cybersecurity company, experienced a significant outage caused by a faulty software update. This incident impacted millions of Windows machines across multiple industries, including transportation, defense, manufacturing, and finance. CrowdStrike has released an official statement and is posting updates on their blog. Microsoft has also published a blog with remediations, which we encourage you to review.

This blog is intended to help existing Splunk customers who are also customers of CrowdStrike gain visibility into how the CrowdStrike outage may be impacting their organizations.

With that in mind, we’d like to share a few detections and additional resources that may help you gain further visibility into how this event is impacting your organization.

Splunk Security Analytics

These detections are designed for use within Splunk Enterprise Security, our SIEM solution that supports our broader unified security and observability platform. For technical support, please reach out to us via the Support Portal.

Find machines with the broken update

This search requires the Splunk Add-on for CrowdStrike, an app that is used with data from Falcon Data Replicator. For more information on the sourcetypes produced by this app, refer to the Splunk documentation.

This search is designed to identify file downloads of the defective CrowdStrike content update on Windows systems by querying CrowdStrike sensor logs. It looks for events where the target file name contains the substring "C-00000291" and ends with ".sys", which might indicate a bad update file, specifically during the LFODownloadConfirmation event. The search is confined to a specific time range that CrowdStrike has provided.

index=* sourcetype="crowdstrike:events:sensor"
TargetFileName="*C-00000291*.sys*" event_simpleName=LFODownloadConfirmation
event_platform=Win earliest=1721362140 latest=1721366820
| stats count min(_time) as firstTime max(_time) as lastTime by aid aip
event_simpleName event_platform TargetFileName DownloadPath
| rename aid as device_id, aip as src_ip , TargetFileName as file_name
| convert timeformat="%Y-%m-%dT%H:%M:%S" ctime(firstTime)
| convert timeformat="%Y-%m-%dT%H:%M:%S" ctime(lastTime)

Sample logs from sourcetype="crowdstrike:events:sensor"

{
    "event_simpleName": "LFODownloadConfirmation",
    "ConfigStateHash": "3600681180",
    "aip": "18.82.155.193",
    "DownloadServer": "lfodown01-b.cloudsink.net",
    "DownloadPath":
 "metahash+/cfs/channelfiles/0000000291/bd9a5c40318869df823085f4449d4c9a/C-00000291-00000000-00000032.sys",
    "DownloadPort": "443",
    "ConfigBuild": "1007.3.0018408.1",
    "event_platform": "Win",
    "Entitlements": "15",
    "name": "LFODownloadConfirmationV1",
    "EventOrigin": "17",
    "CompletionEventId": "Event_ChannelDataDownloadCompleteV1",
    "id": "3aab720a-f12d-139d-883c-693105c5a42c",
    "EffectiveTransmissionClass": "0",
    "aid": "69483d8c3f0bb47758126f30921102e8",
    "timestamp": "1721366804209",
    "cid": "bd9a5c40318869df823085f4449d4c9a",
    "TargetFileName": "C-00000291-00000000-00000032.sys" 
}

This data comes from a production environment. We have anonymized the sensitive fields “aip”, “id”, “cid”, and the “cid” value embedded in the “DownloadPath” field.

Identify machines that have not checked in with CrowdStrike

This search requires the Splunk Add-on for CrowdStrike and the CrowdStrike Devices Inventory for Splunk.

This search identifies Windows machines that have not checked in with CrowdStrike using data from the crowdstrike_devices lookup table and calculates a delta of when it was last seen from the date of the outage July 19, 2024, 04:09:00 UTC.

| inputlookup crowdstrike_devices 
| rex field="category" "os_name: (?[^\|]+)" 
| rex field="category" "dvc_type: (?[^\|]+)" 
| rex field="category" "cs_last_seen: (?[^\|]+)" 
| eval
    cs_last_seen=strptime(cs_last_seen, "%x %T %Z"),
    current_time=now(),
    delta=abs(cs_last_seen - current_time)
| where  match(os, "(?i)windows") AND cs_last_seen>=1721362140 AND cs_last_seen<=1721366820
| convert timeformat="%x %T %Z" ctime(cs_last_seen) ctime(current_time)
| eval delta_days=floor(delta/86400),
        delta_hours=floor((delta%86400)/3600),
        delta_minutes=floor((delta%3600)/60),
        last_checked_in=mvappend(delta_days."d", delta_hours."h", delta_minutes."m")
| eval last_seen=mvjoin(last_checked_in, " ")
| rename nt_host as host, cs_last_seen as last_seen_by_crowdstrike
| table host, mac, ip, os, dvc_type, city, country, last_seen_by_crowdstrike, last_seen
| sort - delta

Additional Apps and Resources for Tracking Host Check-ins

In addition to these detections, there are several apps built by members of the Splunk Community and articles that can help with tracking host check-ins, which is particularly useful in this scenario:

Community-Built Apps on Splunkbase

Articles and Documentation

Splunk Docs: Monitoring Console
Splunk Docs: Deployment Server
Lantern article: Hosts logging data in a certain timeframe
Post from duanewaddle.com: Proving a Negative

Summary

We’ve provided this security content to help Splunk customers gain further visibility into how the CrowdStrike outage may be impacting their organizations. This and the community’s response in general — sharing information, providing solutions, and supporting affected organizations — demonstrates the strength of our industry when faced with unexpected challenges.

While this event serves as a reminder of the potential risks in software updates — especially for products with deployed agents — it also underscores the overall benefits of automatic update processes in maintaining robust security postures.

Moving forward, this incident will likely drive improvements in update deployment and quality assurance processes across the industry. It reinforces the importance of maintaining robust incident response plans and the value of community collaboration in addressing cybersecurity challenges.

Contributors

We would like to thank Kevin Jarvis, Jonathan Heckinger, Emmanuel Jamison, Lou Stella, Bhavin Patel, Rod Soto, Eric McGinnis, Tyne Darke, Anne Ortel and Jose Hernandez for authoring this post, and Zachary Christensen for sharing how to identify machines that have not checked into Splunk.

Splunk Threat Research Team

The Splunk Threat Research Team is an active part of a customer’s overall defense strategy by enhancing Splunk security offerings with verified research and security content such as use cases, detection searches, and playbooks. We help security teams around the globe strengthen operations by providing tactical guidance and insights to detect, investigate and respond against the latest threats. The Splunk Threat Research Team focuses on understanding how threats, actors, and vulnerabilities work, and the team replicates attacks which are stored as datasets in the Attack Data repository.

Our goal is to provide security teams with research they can leverage in their day to day operations and to become the industry standard for SIEM detections. We are a team of industry-recognized experts who are encouraged to improve the security industry by sharing our work with the community via conference talks, open-sourcing projects, and writing white papers or blogs. You will also find us presenting our research at conferences such as Defcon, Blackhat, RSA, and many more.

Staff Picks for Splunk Security Reading August 2022

Check out the latest staff picks from our Splunk security experts, featuring a list of presentations, whitepapers, and customer case studies that we feel are worth a read.

Security 13 Min Read

From Macros to No Macros: Continuous Malware Improvements by QakBot

This blog, the Splunk Threat Research Team (STRT) showcases a year's evolution of QakBot. We also dive into a recent change in tradecraft meant to evade security controls. Last, we reverse engineered the QakBot loader to showcase some of its functions.

Security 10 Min Read

Threat Update: AcidRain Wiper

The Splunk Threat Research Team shares the details on the new malicious payload named AcidRain, designed to wipe modem or router devices (CPEs).

About Splunk

The Splunk platform removes the barriers between data and action, empowering observability, IT and security teams to ensure their organizations are secure, resilient and innovative.

Founded in 2003, Splunk is a global company — with over 7,500 employees, Splunkers have received over 1,020 patents to date and availability in 21 regions around the world — and offers an open, extensible data platform that supports shared data across any environment so that all teams in an organization can get end-to-end visibility, with context, for every interaction and business process. Build a strong data foundation with Splunk.

Learn more about Splunk