Splunk Tools & Analytics To Empower Threat Hunters

Introduction

In the blog post titled “Threat Hunting in 2024: Must-Have Resources & Tasks for Every Hunter,” the SURGe security research team provided an analysis of the daily operations that threat hunters undertake, collected through a survey and interview process. This exploration revealed the most common tasks, resources, and collaboration methods that threat hunters utilize in their day-to-day efforts to protect organizations.

While we at the Splunk Threat Research Team focus on building tools and analytics for threat detection, the insights from the SURGe team’s survey are highly relevant to our work. We want to complement these insights by arming threat hunters with actionable tools and analytics that directly align with the responses from the SURGe survey.

In this blog post, we will:

• Examine two key findings from the survey where we can provide practical support and resources to threat hunters.
• Share practical examples of using Splunk analytics developed by the Splunk Threat Research Team for threat hunting.

By integrating these resources into their threat hunting exercises, we hope to enhance the capability of threat hunters to efficiently identify and mitigate cyber threats before they can cause harm.

Key finding 1: Data

The Bread and Butter of Cyber Defense

One key finding from the threat hunting survey resonated strongly with the Splunk Threat Research Team: “Logs and Other Data'' are identified as the most essential resources for threat hunting, as indicated by survey respondents. Data forms the cornerstone of both threat hunting and detection engineering, although the approaches to log collection may vary between the disciplines.

While threat hunters typically extract their log data directly from production systems to fuel their hunts, our approach for detection development is distinct.

Each detection we write is based on a foundation of simulated log data. Before we can identify patterns that allow us to craft effective detections, it is essential to first understand and simulate the attack techniques in a lab environment.

This approach enhances our detection development capabilities — and it may also benefit threat hunters.

While hunters typically do not generate simulated data themselves, incorporating datasets generated from our simulations of attack techniques could bolster their capabilities. By using these datasets to validate their hunting hypotheses, threat hunters can ensure that their strategies are robust and effectively tuned to real-world attacks.

Attack Data Project

Figure 1.1 Attack Data logo

The Attack Data project is an integral part of our detection development efforts on the Splunk Threat Research Team. This repository houses all the telemetry generated from the simulation of adversary techniques conducted by our team. At the time of writing, the project holds 774 unique datasets totaling ~11GB of event data. The data serves a dual purpose:

• It allows us to develop robust detection analytics.
• It enables us to test our detections as part of our continuous integration and delivery (CI/CD) detection engineering processes.

What’s in it for threat hunters: Threat hunters can leverage the Attack Data project to validate their hunting hypotheses against a comprehensive set of simulated attack data. By accessing and utilizing these datasets, they can test the effectiveness of their strategies and refine their approaches to better detect and mitigate real-world threats.

Each dataset within the repository is organized according to a consistent structure, outlined in a YAML file that accompanies the data. These YAML files include several key fields including:

• ID: A unique identifier for the dataset
• Author: The creator of the dataset
• Date: The last modified date
• Description: Detailed information about what the dataset simulates
• Environment: The setup in which the dataset was collected
• Technique: The MITRE ATT&CK techniques that the dataset relates to
• Dataset: Links to the hosted versions of the dataset
• References: Links to resources that reference the dataset
• Sourcetypes: The Splunk sourcetype included in the dataset

Figure 1.2 Dataset YAML example. Source

To illustrate the practical use of the Attack Data project for threat hunters, let's walk through an example of how a dataset can be replayed into a Splunk instance to test and refine threat hunting strategies.

1. Download the Dataset: Start by visiting the Attack Data repository on GitHub. Select a dataset that corresponds to a specific attack technique you wish to test.
2. Import the Dataset into Splunk: To import the dataset, you can either use our replay.py script for automated ingestion or manually upload the data through Splunk’s 'Add Data' feature in the UI.
3. Query and Analyze: Once the data is imported, use Splunk search and analysis tools to run queries against the data.
4. Refine Your Hunts: Based on the outcomes of your queries, you may identify adjustments to make to your hunting analytics rules.
5. Iterate and Validate: Continue this process with different datasets and attack techniques to ensure your threat hunting capabilities are robust and can handle various threat scenarios.

Key finding 2: Data Analysis

Finding the Needle in the Cyber Haystack

“Analyzing logs” was highlighted as the most common essential task by threat hunters in the survey. This activity is at the heart of what the Splunk Threat Research Team does.

We support and maintain the Enterprise Security Content Update (ESCU) app, which packages security content in the form of Splunk detection analytics and SOAR playbooks. Currently, the app comprises over 1,600 analytics that span numerous data sources and address a wide range of cyber threat techniques.

Visit research.splunk.com to view the Splunk Threat Research Team's complete security content repository.

Figure 1.3 Security Content Overview

While many of the analytics in the ESCU app are designed for signature-based detection, focusing on specific behaviors not suited for threat hunting, some can also be used to assist hunters in sifting through data, providing extra analysis tools to search for and identify potential security threats.

What’s in it for threat hunters: To allow hunters to identify the most suitable analytics for their needs, they are grouped into several functional types, each designed to enhance specific aspects of threat detection and response.

Understanding the various types of analytics available in the ESCU app can help threat hunters choose the most appropriate tools for their specific challenges.

Figure 1.4 ESCU Analytics by Type

The “Anomaly” and “Hunting” analytic types are particularly well-suited for threat hunters, since they focus on:

• Uncovering unusual patterns.
• Increasing asset risk scoring.

Additionally, many of the TTP (tactics, techniques, and procedures) analytics, which are designed to detect specific adversarial behaviors, also serve as valuable tools for hunting. These TTP analytics provide a robust framework for threat hunters, enabling them to identify and analyze well-documented attack methodologies effectively.

At the time of writing, the Splunk Threat Research Team has built over 252 hunting type detections across various data sources. Below is a breakdown per data source of the out-of-the-box hunting detections you can take advantage of today in Splunk Enterprise Security. 

Figure 1.5 ESCU Hunting Detections By Data Source

Practical examples: Hunting analytics

The next part of this blog explores practical examples of hunting analytics from the ESCU app, focusing on the top data sources. Our goal is to showcase how ESCU can empower threat hunters with effective analysis tools across various platforms. To provide a clear and concise overview, we’ll highlight one significant analytic per data source:

• PowerShell Script Block Hunting for endpoint data
• Azure AD Multi-Source Failed Authentications Spike for cloud data
• Okta MFA Exhaustion Hunt for application data

Endpoint data

Endpoint monitoring remains a critical front for defense against adversaries. Endpoints serve as both entry points and battlegrounds in the fight against malicious activities, making their monitoring essential for early detection and mitigation of threats.

By keeping a vigilant watch over endpoint activities, threat hunters can proactively search for and identify unusual behaviors that may signify an attack in progres.

A powerful scripting tool embedded in Windows, PowerShell continues to be a double-edged sword. While it offers administrators and users vast capabilities for automation and management, it has also become a favored tool for attackers. By abusing PowerShell, adversaries can execute code remotely, escalate privileges, and extract sensitive information, often bypassing traditional security measures due to PowerShell’s legitimate use within corporate environments.

For more context and a hands-on demo on the topic, we invite you to watch the Hunting for Malicious PowerShell using Script Block Logging Splunk Tech Talk.

PowerShell Script Block Hunting

In the context of endpoint security, monitoring PowerShell usage is crucial due to its powerful capabilities, which can be abused by attackers. Our PowerShell Script Block Logging hunting analytic specifically targets the execution of potentially malicious scripts. Key metrics analyzed by this analytic include:

• Frequency and patterns of suspicious script blocks
• Use of uncommon or malicious/suspicious imported PowerShell cmdlets
• Inclusion of suspicious keywords
• Attempts to execute encoded commands

By scrutinizing these elements, threat hunters can detect potential malicious PowerShell activity, tailoring thresholds to capture both overt and subtle threats effectively.

Additionally, the analytic employs a scoring system where each detected behavior is assigned a score reflecting its potential risk. These scores are then aggregated to calculate a total score, stored in a new field in the analytic. This field allows for easy sorting and prioritization of results by score in Splunk, streamlining the process of identifying the most critical threats to investigate first.

`powershell` EventCode=4104
| eval DoIt = if(match(ScriptBlockText,"(?i)(\$doit)"), "4", 0)
| eval enccom=if(match(ScriptBlockText,"[A-Za-z0-9+\/]{44,}([A-Za-z0-9+\/]{4}
|[A-Za-z0-9+\/]{3}=
|[A-Za-z0-9+\/]{2}==)") OR match(ScriptBlockText, "(?i)[-]e(nc*o*d*e*d*c*o*m*m*a*n*d*)*\s+[^-]"),4,0)
| eval suspcmdlet=if(match(ScriptBlockText, "(?i)Add-Exfiltration |Add-Persistence |Add-RegBackdoor |Add-ScrnSaveBackdoor |Check-VM |Do-Exfiltration |Enabled-DuplicateToken |Exploit-Jboss |Find-Fruit |Find-GPOLocation |Find-TrustedDocuments |Get-ApplicationHost |Get-ChromeDump |Get-ClipboardContents |Get-FoxDump |Get-GPPPassword |Get-IndexedItem |Get-Keystrokes |LSASecret |Get-PassHash |Get-RegAlwaysInstallElevated |Get-RegAutoLogon |Get-RickAstley |Get-Screenshot |Get-SecurityPackages |Get-ServiceFilePermission |Get-ServicePermission |Get-ServiceUnquoted |Get-SiteListPassword |Get-System |Get-TimedScreenshot |Get-UnattendedInstallFile |Get-Unconstrained |Get-VaultCredential |Get-VulnAutoRun |Get-VulnSchTask |Gupt-Backdoor |HTTP-Login |Install-SSP |Install-ServiceBinary |Invoke-ACLScanner |Invoke-ADSBackdoor |Invoke-ARPScan |Invoke-AllChecks |Invoke-BackdoorLNK |Invoke-BypassUAC |Invoke-CredentialInjection |Invoke-DCSync |Invoke-DllInjection |Invoke-DowngradeAccount |Invoke-EgressCheck |Invoke-Inveigh |Invoke-InveighRelay |Invoke-Mimikittenz |Invoke-NetRipper |Invoke-NinjaCopy |Invoke-PSInject |Invoke-Paranoia |Invoke-PortScan |Invoke-PoshRat |Invoke-PostExfil |Invoke-PowerDump |Invoke-PowerShellTCP |Invoke-PsExec |Invoke-PsUaCme |Invoke-ReflectivePEInjection |Invoke-ReverseDNSLookup |Invoke-RunAs |Invoke-SMBScanner |Invoke-SSHCommand |Invoke-Service |Invoke-Shellcode |Invoke-Tater |Invoke-ThunderStruck |Invoke-Token |Invoke-UserHunter |Invoke-VoiceTroll |Invoke-WScriptBypassUAC |Invoke-WinEnum |MailRaider |New-HoneyHash |Out-Minidump |Port-Scan |PowerBreach |PowerUp |PowerView |Remove-Update |Set-MacAttribute |Set-Wallpaper |Show-TargetScreen |Start-CaptureServer |VolumeShadowCopyTools |NEEEEWWW  |(Computer |User)Property |CachedRDPConnection |get-net\S+ |invoke-\S+hunter |Install-Service |get-\S+(credent |password) |remoteps |Kerberos.*(policy |ticket) |netfirewall |Uninstall-Windows |Verb\s+Runas |AmsiBypass |nishang |Invoke-Interceptor |EXEonRemote |NetworkRelay |PowerShelludp |PowerShellIcmp |CreateShortcut |copy-vss |invoke-dll |invoke-mass |out-shortcut |Invoke-ShellCommand"),1,0)
| eval base64 = if(match(lower(ScriptBlockText),"frombase64"), "4", 0)
| eval empire=if(match(lower(ScriptBlockText),"system.net.webclient") AND match(lower(ScriptBlockText), "frombase64string") ,5,0)
| eval mimikatz=if(match(lower(ScriptBlockText),"mimikatz") OR match(lower(ScriptBlockText), "-dumpcr") OR match(lower(ScriptBlockText), "SEKURLSA::Pth") OR match(lower(ScriptBlockText), "kerberos::ptt") OR match(lower(ScriptBlockText), "kerberos::golden") ,5,0)
| eval iex=if(match(ScriptBlockText, "(?i)iex
|invoke-expression"),2,0)
| eval webclient=if(match(lower(ScriptBlockText),"http") OR match(lower(ScriptBlockText),"web(client
|request)") OR match(lower(ScriptBlockText),"socket") OR match(lower(ScriptBlockText),"download(file
|string)") OR match(lower(ScriptBlockText),"bitstransfer") OR match(lower(ScriptBlockText),"internetexplorer.application") OR match(lower(ScriptBlockText),"xmlhttp"),5,0)
| eval get = if(match(lower(ScriptBlockText),"get-"), "1", 0)
| eval rundll32 = if(match(lower(ScriptBlockText),"rundll32"), "4", 0)
| eval suspkeywrd=if(match(ScriptBlockText, "(?i)(bitstransfer |mimik |metasp |AssemblyBuilderAccess |Reflection\.Assembly |shellcode |injection |cnvert |shell\.application |start-process |Rc4ByteStream |System\.Security\.Cryptography |lsass\.exe |localadmin |LastLoggedOn |hijack |BackupPrivilege |ngrok |comsvcs |backdoor |brute.?force |Port.?Scan |Exfiltration |exploit |DisableRealtimeMonitoring |beacon)"),1,0)
| eval syswow64 = if(match(lower(ScriptBlockText),"syswow64"), "3", 0)
| eval httplocal = if(match(lower(ScriptBlockText),"http://127.0.0.1"), "4", 0)
| eval reflection = if(match(lower(ScriptBlockText),"reflection"), "1", 0)
| eval invokewmi=if(match(lower(ScriptBlockText), "(?i)(wmiobject |WMIMethod |RemoteWMI |PowerShellWmi |wmicommand)"),5,0)
| eval downgrade=if(match(ScriptBlockText, "(?i)([-]ve*r*s*i*o*n*\s+2)") OR match(lower(ScriptBlockText),"powershell -version"),3,0)
| eval compressed=if(match(ScriptBlockText, "(?i)GZipStream
|::Decompress
|IO.Compression
|write-zip
|(expand
|compress)-Archive"),5,0)
| eval invokecmd = if(match(lower(ScriptBlockText),"invoke-command"), "4", 0)
| addtotals fieldname=Score DoIt, enccom, suspcmdlet, suspkeywrd, compressed, downgrade, mimikatz, iex, empire, rundll32, webclient, syswow64, httplocal, reflection, invokewmi, invokecmd, base64, get
| stats values(Score) by UserID, Computer, DoIt, enccom, compressed, downgrade, iex, mimikatz, rundll32, empire, webclient, syswow64, httplocal, reflection, invokewmi, invokecmd, base64, get, suspcmdlet, suspkeywrd
| rename Computer as dest, UserID as user

Figure 1.6 PowerShell Script Block Hunting

Cloud data

Azure Active Directory (now called Microsoft Entra ID) serves as Microsoft’s cloud-based identity and access management service, underpinning the authentication for Microsoft 365 and numerous other applications. As the authentication entry point for Microsoft’s cloud services, hunting in Azure AD logs is crucial for protecting a vast array of enterprise and personal data.

As cloud adoption continues to surge, identity has increasingly become the new perimeter in cybersecurity. In this landscape, it's crucial for threat hunters to focus on identifying signs of account takeover attacks, a prevalent threat in cloud environments. Password spraying remains a prevalent threat and adversaries continue to leverage this attack vector in the wild.

The Microsoft Digital Defense Report of 2023 highlights a surge in password-based attacks against the Microsoft platform, with incidents of password spraying increasing more than tenfold. The same report documents sophisticated password spray campaigns that use a distributed network of IP addresses across various countries to evade security controls and account lockouts.

For more details and a hands-on demo on the topic, we invite you to watch the Purple Teaming to Enhance Detection Engineering SANS webinar.

Azure AD Multi-Source Failed Authentications Spike

Hunting for a multi-source distributed password spray requires a robust strategy. Our approach hinges on identifying authentication spikes within a short period of time that exhibit specific characteristics. Our Azure AD Multi-Source Failed Authentications Spike analytic calculates key metrics like:

• Number of unique user-IP combinations
• Count of distinct users
• Diversity of source IPs and countries
• Unique number of user-agents involved in failed login attempts.

Calculating the number of unique user-IP combinations allows us to highlight unusual patterns of failed logins that may indicate a distributed password spray attack. By customizing the thresholds for the calculated metrics, threat hunters can tailor the hunt analytic to detect both more obvious and stealthy attacks, adapting the sensitivity according to the specifics of their security environment.

`azure_monitor_aad` category=SignInLogs properties.status.errorCode=50126 properties.authenticationDetails{}.succeeded=false
 | rename properties.* as *
 | bucket span=5m _time
 | eval uniqueIPUserCombo = src_ip . "-" . user
 | stats count min(_time) as firstTime max(_time) as lastTime dc(uniqueIPUserCombo) as uniqueIpUserCombinations, dc(user) as uniqueUsers, dc(src_ip) as uniqueIPs, dc(user_agent) as uniqueUserAgents, dc(location.countryOrRegion) as uniqueCountries values(user) as users, values(src_ip) as ips, values(user_agent) as user_agents, values(location.countryOrRegion) as countries
 | where uniqueIpUserCombinations > 20 AND uniqueUsers > 20 AND uniqueIPs > 20 AND uniqueUserAgents = 1

Figure 1.7 Azure AD Multi-Source Failed Authentications Spike

Application data

Okta is a widely used cloud-based identity management service that provides single sign-on and multi-factor authentication (MFA) solutions to secure user access across various applications.

As organizations increasingly rely on such services for enhanced security, the prevalence of MFA fatigue attacks has grown. These attacks exploit human factors by bombarding users with MFA requests until they inadvertently approve a malicious login, leading to significant breaches.

Threat hunters should remain vigilant against MFA fatigue attacks and proactively perform hunts that aim to identify potential victims or detect signs of breaches early.

Okta MFA Exhaustion Hunt

Hunting for signs of MFA fatigue involves a strategic analysis of authentication attempts. Our Okta-focused hunting analytic is designed to detect spikes in failed MFA attempts that exhibit unusual patterns over a short period of time. This analytic assesses critical metrics including:

• Number of MFA push notifications sent
• Counts of successful and failed attempts
• Frequency of these attempts per user
• Duration between the first and last MFA attempt

This query uniquely analyzes how MFA request patterns cluster within short time windows, allowing threat hunters to spot potential MFA fatigue situations. By observing the sequence of repeated failed attempts followed by a sudden success, the analytic can indicate a user succumbing to fatigue and inadvertently approving a malicious login.

By setting customizable thresholds for these metrics, threat hunters can fine-tune the analytic to detect both blatant and subtle signs of MFA fatigue, adapting the sensitivity to meet the specific security needs of their environment.

`okta` eventType=system.push.send_factor_verify_push OR ((legacyEventType=core.user.factor.attempt_success) AND (debugContext.debugData.factor=OKTA_VERIFY_PUSH)) OR ((legacyEventType=core.user.factor.attempt_fail) AND (debugContext.debugData.factor=OKTA_VERIFY_PUSH))
  | stats count(eval(legacyEventType="core.user.factor.attempt_success"))  as successes count(eval(legacyEventType="core.user.factor.attempt_fail")) as failures count(eval(eventType="system.push.send_factor_verify_push")) as pushes by user,_time
  | stats latest(_time) as lasttime earliest(_time) as firsttime sum(successes) as successes sum(failures) as failures sum(pushes) as pushes by user
  | eval seconds=lasttime-firsttime | eval lasttime=strftime(lasttime, "%c")
  | search (pushes>1)
  | eval totalattempts=successes+failures
  | eval finding="Normal authentication pattern"
  | eval finding=if(failures==pushes AND pushes>1,"Authentication attempts not successful because multiple pushes denied",finding)
  | eval finding=if(totalattempts==0,"Multiple pushes sent and ignored",finding)
  | eval finding=if(successes>0 AND pushes>3,"Probably should investigate. Multiple pushes sent, eventual successful authentication!",finding)

Figure 1.8 Okta MFA Exhaustion Hunt

Closing Thoughts

Throughout this blog post, we've delved into insights from the SURGe team's survey and highlighted how tools and resources developed by the Splunk Threat Research Team can assist threat hunters. By aligning these tools with the real-world needs of threat hunters, we strive to advance cybersecurity practices across various domains.

As we continue to refine our approaches and develop new analytics, we encourage threat hunters to leverage these resources in their ongoing efforts to secure their environments against increasingly sophisticated threats.

For additional perspectives on threat hunting, the Cisco Talos research team provides valuable insights into the proactive strategies that can be employed to uncover hidden threats. You can learn more about their approach here.

Learn More

Visit research.splunk.com to view the Splunk Threat Research Team's complete security content repository. You can implement this content using the Enterprise Security Content Updates app or the Splunk Security Essentials app.

We would like to thank Mauricio Velazco for authoring this post and the entire Splunk Threat Research Team for their contributions.