Introducing ShellSweepPlus: Open-Source Web Shell Detection

Today, the Splunk Threat Research Team is thrilled to introduce ShellSweepPlus, an advancement in our ongoing mission to combat the persistent threat of web shells.

Building upon the solid foundation of its predecessor ShellSweep, ShellSweepPlus is an enhanced version that takes web shell detection to new heights, incorporating cutting-edge techniques and a multifaceted approach to safeguard your web environments.

ShellSweepPlus Overview

ShellSweepPlus is an open-source tool designed to empower security teams in detecting potential web shells. It is an enhanced version of ShellSweep

The threat of web shells

Web shells pose a significant threat to organizations, as they provide attackers with unauthorized access and control over compromised web servers. Attackers can exploit these shells to:

• Steal sensitive data.
• Deploy additional malware.
• Launch further attacks.

Detecting potential web shells promptly is crucial for minimizing the impact of a breach and preventing further damage.

How ShellSweepPlus augments threat defense

ShellSweepPlus provides immense value to security teams by enhancing the impact of existing security tools.

By integrating ShellSweepPlus into their security stack, organizations can augment their defenses and gain a more comprehensive view of their web application security posture. The tool's advanced detection methods — such as entropy analysis, pattern matching, and heuristic analysis —enable security professionals to identify potential web shells with high accuracy, even if they are heavily obfuscated or employ novel techniques.

Moreover, ShellSweepPlus empowers security teams to identify threats that might otherwise go unnoticed. Its detailed reporting and alerting capabilities enable swift response to potential web shell threats, reducing the time between detection and remediation.

By proactively hunting for anomalies and suspicious files, security professionals can stay one step ahead of attackers and bolster their organization's overall security resilience.

Whether you are a seasoned security expert or new to the field, ShellSweepPlus is designed to be an indispensable, modular and configurable tool. Its versatility, accessibility, and powerful features make it a valuable tool in the fight against the persistent threat of web shells.

ShellSweepPlus: Key Enhancements and Features

With ShellSweepPlus, we have introduced several enhancements that bolster its detection capabilities:

1. Heuristic Analysis: ShellSweepPlus introduces the `Perform-HeuristicAnalysis` function, which employs advanced heuristic rules to identify anomalies and detect even the most elusive web shells. This addition greatly enhances the tool's ability to uncover previously unknown threats.
2. Enhanced Pattern Detection: The `Detect-WebshellPatterns` function has been implemented to identify known malicious patterns, providing an additional layer of protection. By leveraging a comprehensive set of suspicious patterns, ShellSweepPlus can quickly flag potential web shell files.
3. Entropy and Standard Deviation Integration: ShellSweepPlus excels at computing the mean and standard deviation of entropy values for each file extension. Files that deviate significantly from the expected entropy range are flagged, enabling precise and comprehensive detection.
4. Mixed-Mode Detection: ShellSweepPlus employs a synergistic blend of entropy-based detection, standard deviation-based detection, and mixed-mode detection (utilizing standard deviation with hardcoded thresholds). This multifaceted strategy ensures thorough coverage and enhanced reliability in identifying potential web shells.
5. Static Code Analysis: The tool incorporates a sophisticated pattern-based detection mechanism, analyzing code statically to identify and flag known malicious patterns.

While ShellSweepPlus introduces these advanced features, we recognize the importance of baselining and truly understanding your environment. Tools like ShellCSV and ShellScan, which emphasize baselining, remain important companions to ShellSweepPlus, providing visibility into your system's behavior.

As we delve into this exciting new chapter with ShellSweepPlus, join us in exploring its enhanced capabilities and discover how it can enhance your web shell detection efforts.

New Features of ShellSweepPlus

Before we dive into the features of ShellSweepPlus, it is important to remember ShellSweep is a suite of tools that work together.

1. First, baseline your server with ShellScan to update the entropy values.
2. Tune ShellSweepPlus to your servers by modifying the extensions and entropy values.
3. Last, setup ShellSweep|Plus to run.

These steps were outlined in detail in the original ShellSweep blog.

(ShellSweep flow diagram, Splunk 2024)

Now, let’s check out the new features we’ve added.

Keywords Analysis

One of the initial methods we sought to incorporate was keyword matching. To achieve this, we crafted a PowerShell script designed to scan numerous web shell directories, examining the words within the files.

Our primary goal was to ensure that we only included keywords that consistently appeared across multiple web shells. We set a threshold of three, meaning that a word had to appear in three or more files to be added to the hashtable and subsequently output.

Below is the script we used. It’s also available in the ShellSweep repository.

# This script is used to extract keywords from a set of directories containing webshell files.
# It reads each file, splits the content into words and updates each word's frequency in a hash table.
# It then filters out words that appear more than 3 times and considers them as suspicious.
# The suspicious words are then written to a file 'suspiciousPatterns.txt'.
 
 
$webshellDirectoryPath = @(
'C:\Users\Administrator\Downloads\reGeorg-master\reGeorg-master',
'C:\Users\Administrator\Downloads\p0wny-shell-master',
'C:\Users\Administrator\Desktop\10684728197_human2_cisa_report',
'C:\Users\Administrator\Downloads\xl7dev\WebShell-master',
'C:\Users\Administrator\Downloads\webshells-master\webshells-master',
'C:\Users\Administrator\Downloads\webshell-master\webshell-master',
'C:\Users\Administrator\Desktop\10660311902'
)
 
$wordFrequencyInDirectory = @{}
 
# Walk through each file in the directory
Get-ChildItem $webshellDirectoryPath -File | foreach {
$content = Get-Content $_.FullName -Raw
 
# Split the content into words and update each word's frequency in the hash table
$content -split '\s+' | foreach {
if ($wordFrequencyInDirectory.ContainsKey($_)) {
$wordFrequencyInDirectory[$_]++
} else {
$wordFrequencyInDirectory.Add($_, 1)
}
}
}
 
# Filter out words that appear more than 3 times
$suspiciousWords = $wordFrequencyInDirectory.GetEnumerator() | Where-Object { $_.Value -gt 3 } | ForEach-Object { $_.Key }
 
$output = "`$suspiciousPatterns = @(" + "`r`n"
foreach ($word in $suspiciousWords) {
$output += " '$word'," + "`r`n"
}
$output += ")"
 
$output | Out-File -FilePath 'C:\temp\suspiciousPatterns.txt'

(Generated Suspicious Keywords, Splunk 2024)

Here is the output of that script: a hash table that we can accept and drop right into ShellSweepPlus.

Static Code Analysis

In ShellSweepPlus, we have introduced a powerful static code analysis feature that enhances the tool's ability to identify potential web shells. This feature uses a set of predefined suspicious patterns ($suspiciousPatterns) to perform in-depth pattern-based detections, allowing ShellSweepPlus to catch even the most elusive threats.

The static code analysis is implemented through the Detect-WebshellPatterns function, which scans the content of each file and matches it against the suspicious patterns. These patterns can include common web shell keywords, functions, or code snippets that are often associated with malicious activities.

Here's a closer look at the Detect-WebshellPatterns function:

function Detect-WebshellPatterns {
    param(
        [Parameter(Mandatory=$true)] [string] $FileContent
    )
 
    $matchedPatterns = @()
 
    foreach ($pattern in $suspiciousPatterns) {
        if ($FileContent -match $pattern) {
            $matchedPatterns += $pattern
        }
    }
 
    return $matchedPatterns
}

The function takes the file content as input and iterates over each suspicious pattern defined in the $suspiciousPatterns array. If a pattern is found within the file content, it is added to the $matchedPatterns array. Finally, the function returns the list of matched patterns.

You can easily customize and extend the suspicious patterns to include additional patterns specific to your environment or newly discovered web shell indicators. This flexibility allows ShellSweepPlus to:

• Adapt to evolving threats.
• Maintain its effectiveness in detecting web shells.

When a file is processed by ShellSweepPlus, the Detect-WebshellPatterns function is called to identify any suspicious patterns within the file content. If matches are found, they are included in the result object, providing valuable information about the potential web shell.

Here's an example of how the static code analysis is integrated into the file processing:

function Process-File {
    param(
        [hashtable] $file,
        [string] $detectionMethod,
        [double] $baseScore,
        [double] $sdForExt = $null
    )
 
    # ...
 
    $matchedPatterns = Detect-WebshellPatterns -FileContent $content
    $confidenceScore = Adjust-ConfidenceScore -baseScore $baseScore -matchedPatterns $matchedPatterns
 
    $result = Create-ResultObject -path $path -entropy $entropy -sdForExt $sdForExt -hash $hash -lastModified $lastModified -detectionMethod $detectionMethod -confidenceScore $confidenceScore -matchedPatterns $matchedPatterns
 
    $result | ConvertTo-Json -Compress
}

In the Process-File function, Detect-WebshellPatterns function is called with the file content, and the matched patterns are stored in the$matchedPatterns variable. These matched patterns are then used to adjust the confidence score and are included in the result object.

The static code analysis feature in ShellSweepPlus adds an extra layer of detection capabilities, complementing the entropy-based and standard deviation-based detection methods. By identifying suspicious patterns within the file content, ShellSweepPlus can flag potential web shells that may have evaded other detection techniques.

This powerful combination of static code analysis and other detection methods makes ShellSweepPlus a comprehensive tool for identifying and mitigating the risk of web shells in your environment.

Entropy and Standard Deviation Integration

In ShellSweepPlus, we have introduced a powerful feature that leverages the statistical concept of standard deviation to enhance our entropy-based detection capabilities.

Standard deviation is a measure of the dispersion of a set of values from their mean. In the context of entropy analysis, it helps us identify files that deviate significantly from the expected entropy range for a given file extension.

If you're familiar with Splunk's stdev function or have encountered our content that uses it, you'll be pleased to know that we have incorporated a similar approach directly into our PowerShell script. By calculating the standard deviation of entropy values for each file extension, we can establish a baseline and flag files that exhibit anomalous entropy levels.

The implementation of this feature in ShellSweepPlus is as follows:

# This hashtable will store the mean entropy and standard deviation for each file extension
$statsPerExtension = @{}
 
# Calculate mean and standard deviation for each extension
foreach ($extension in $entropiesPerExtension.Keys) {
    $entropies = $entropiesPerExtension[$extension]
 
    # Compute the mean entropy
    Write-Verbose "Calculating mean entropy for $extension files..."
    $meanEntropy = ($entropies | Measure-Object -Sum).Sum / $entropies.Count
 
    # Compute the squared differences from the mean
    $squaredDifferences = $entropies | ForEach-Object {
        $difference = $_ - $meanEntropy
        return [Math]::Pow($difference, 2)
    }
 
    # Compute the mean of the squared differences
    $meanOfSquaredDifferences = ($squaredDifferences | Measure-Object -Sum).Sum / $squaredDifferences.Count
 
    # Compute the standard deviation for entropy
    Write-Verbose "Calculating standard deviation for entropy of $extension files..."
    $sdEntropy = [Math]::Sqrt($meanOfSquaredDifferences)
 
    $statsPerExtension[$extension] = @{
        'Mean' = $meanEntropy
        'StandardDeviation' = $sdEntropy
    }
}

Let's break down the steps involved in this process:

1. We start by initializing a hashtable called $statsPerExtension to store the mean entropy and standard deviation for each file extension.
2. We iterate over each file extension in the $entropiesPerExtension hashtable, which contains the entropy values for files grouped by their extension.
3. For each file extension, we calculate the mean entropy by summing up all the entropy values and dividing by the count of files with that extension.
4. Next, we compute the squared differences between each entropy value and the mean entropy. This step is necessary for calculating the standard deviation.
5. We calculate the mean of the squared differences by summing them up and dividing by the count of files.
6. Finally, we compute the standard deviation for the entropy values of each file extension by taking the square root of the mean of squared differences.
7. The mean entropy and standard deviation for each file extension are stored in the $statsPerExtension hashtable for later use.

By incorporating this entropy and standard deviation integration, ShellSweepPlus gains a more nuanced and adaptive approach to detecting anomalous files. It allows us to identify files that exhibit entropy values significantly different from the norm for their respective file extensions, providing a more robust and reliable detection mechanism.

Heuristic Analysis

ShellSweepPlus takes web shell detection to the next level with the introduction of the `Perform-HeuristicAnalysis` function.

This advanced feature employs sophisticated heuristic rules to identify anomalies and detect even the most elusive zero-day web shells, greatly enhancing the tool's ability to uncover previously unknown threats.

The heuristic analysis is performed by the `Perform-HeuristicAnalysis` function, which scans the content of each file and matches it against a predefined set of heuristic rules (`$heuristicRules`). These rules are designed to identify patterns, techniques, or behaviors that are commonly associated with web shells but may not be captured by traditional signature-based detection methods.

Here's a closer look at the `Perform-HeuristicAnalysis` function:

function Perform-HeuristicAnalysis {
        param(
            [Parameter(Mandatory=$true)] [string] $FileContent
        )
     
        $matchedHeuristics = @()
     
        foreach ($rule in $heuristicRules) {
            if ($FileContent -match $rule.pattern) {
                $matchedHeuristics += $rule.description
            }
        }
     
        return $matchedHeuristics
    }
     

Here are some out-of-the-box heuristic rules that leverage `Perform-HeuristicAnalysis` function:

$heuristicRules = @(
    @{ 'pattern' = 'eval\('; 'description' = 'Use of eval function' },
    @{ 'pattern' = 'base64_decode\('; 'description' = 'Use of base64_decode function' },
    @{ 'pattern' = 'shell_exec\('; 'description' = 'Use of shell_exec function' },
    @{ 'pattern' = 'proc_open\('; 'description' = 'Use of proc_open function' },
    @{ 'pattern' = 'popen\('; 'description' = 'Use of popen function' },
    @{ 'pattern' = 'passthru\('; 'description' = 'Use of passthru function' },
…
    @{ 'pattern' = 'register_tick_function\('; 'description' = 'Use of register_tick_function' },
    @{ 'pattern' = 'ob_start\('; 'description' = 'Use of ob_start function' },
    @{ 'pattern' = 'ob_get_contents\('; 'description' = 'Use of ob_get_contents function' },
    @{ 'pattern' = 'ob_get_clean\('; 'description' = 'Use of ob_get_clean function' },
    @{ 'pattern' = 'ob_end_clean\('; 'description' = 'Use of ob_end_clean function' },
    @{ 'pattern' = 'ob_flush\('; 'description' = 'Use of ob_flush function' },
    @{ 'pattern' = 'ob_end_flush\('; 'description' = 'Use of ob_end_flush function' }
)

The function takes the file content as input and iterates over each heuristic rule defined in the `$heuristicRules` array. Each rule consists of a pattern and a description. If a pattern is found within the file content, the corresponding description is added to the `$matchedHeuristics` array. Finally, the function returns the list of matched heuristics.

You can easily expand and customize the heuristic rules based on new findings, emerging threats, or specific characteristics of web shells observed in your environment. This flexibility ensures that ShellSweepPlus remains effective in detecting even the most sophisticated and evasive web shells.

When a file is processed by ShellSweepPlus, the `Perform-HeuristicAnalysis` function is called to identify any anomalies or suspicious behaviors within the file content. If matches are found, they are included in the result object, providing valuable insights into the potential web shell.

The heuristic analysis feature in ShellSweepPlus adds an extra layer of intelligence to the detection process, complementing the other detection methods. By identifying anomalies and suspicious patterns that may not be covered by traditional signatures, ShellSweepPlus significantly enhances its ability to detect zero-day web shells and previously unknown threats.

Mixed-Mode Detection

ShellSweepPlus takes a multifaceted approach to web shell detection by employing a synergistic blend of entropy-based detection, standard deviation-based detection, and mixed-mode detection. This comprehensive strategy ensures thorough coverage and enhanced reliability in identifying potential web shells.

The mixed-mode detection feature in ShellSweepPlus combines the strengths of entropy-based and standard deviation-based detection methods with the added benefit of hardcoded thresholds. By utilizing multiple detection techniques and combining their results, ShellSweepPlus achieves a higher level of accuracy and effectiveness in identifying web shell threats.

Here's how the mixed-mode detection is implemented in ShellSweepPlus:

# Detect anomalies based on standard deviation modified thresholds but also hard coded GT value
Write-Verbose "Performing mixed mode detection..."
foreach ($file in $allFiles) {
    $extension = [System.IO.Path]::GetExtension($file['Path'])
    $sdForExt = $statsPerExtension[$extension]['StandardDeviation']
    Process-File -file $file -detectionMethod "Mixed Mode" -baseScore $weights["Mixed Mode"] -sdForExt $sdForExt
}

In the mixed-mode detection loop, ShellSweepPlus iterates over each file and performs the following steps:

1. It retrieves the file extension and the corresponding standard deviation value from the `$statsPerExtension` hashtable.
2. It calls the `Process-File` function, passing the file, detection method ("Mixed Mode"), base score (`$weights["Mixed Mode"]`), and the standard deviation value for the file extension.

The `Process-File` function then applies the mixed-mode detection logic to the file:

function Process-File {
        param(
            [hashtable] $file,
            [string] $detectionMethod,
            [double] $baseScore,
            [double] $sdForExt = $null
        )
     
        # ...
     
        $matchedPatterns = Detect-WebshellPatterns -FileContent $content
        $confidenceScore = Adjust-ConfidenceScore -baseScore $baseScore -matchedPatterns $matchedPatterns
     
        $result = Create-ResultObject -path $path -entropy $entropy -sdForExt $sdForExt -hash $hash -lastModified $lastModified -detectionMethod $detectionMethod -confidenceScore $confidenceScore -matchedPatterns $matchedPatterns
     
        $result | ConvertTo-Json -Compress
    }

Inside the `Process-File` function, the mixed-mode detection performs the following steps:

1. It calls the `Detect-WebshellPatterns` function to identify any known malicious patterns within the file content.
2. It adjusts the confidence score based on the matched patterns using the `Adjust-ConfidenceScore` function.
3. It creates a result object using the `Create-ResultObject` function, including the file path, entropy, standard deviation, hash, last modified date, detection method, confidence score, and matched patterns.
4. Finally, it converts the result object to JSON format and outputs it.

The mixed-mode detection leverages the standard deviation value for the file extension to determine if the file's entropy falls within the expected range. It also utilizes hardcoded thresholds defined in the `$fileExtensions` hashtable to further refine the detection process.

By combining entropy-based detection, standard deviation-based detection, and hardcoded thresholds, the mixed-mode detection feature in ShellSweepPlus provides a robust and reliable approach to identifying potential web shells. It takes into account multiple factors and detection techniques to make informed decisions about the likelihood of a file being a web shell.

The mixed-mode detection, along with other advanced features in ShellSweepPlus like heuristic analysis and enhanced pattern detection, creates a comprehensive and effective solution for detecting and mitigating the risk of web shells in your environment.

How To Implement ShellSweepPlus

ShellSweepPlus may be implemented in a stand alone fashion or utilized with the Splunk Universal Forwarder. Below we will highlight these two methods.

Implement with Splunk

As with ShellSweep, it’s fairly easy to get going and deployed to web servers across the fleet.

First, find a home for ShellSweepPlus or create a new App. Within the App, two items are required:

• inputs.conf
• ShellSweepPlus.ps1 Script in the Bin directory

Add the script to the inputs.conf, which will run the script every day at midnight. Modify as needed based on requirements.

[powershell://ShellSweepPlus]
script = . "$SplunkHome\etc\apps\win_inputs_app\bin\ShellSweepPlus.ps1"
disabled = false
sourcetype = shellsweepplus
schedule = 0 0 * * *
index = win

Now, add ShellSweepPlus to the bin src. The latest version is located here. By default, ShellSweep will output to JSON, allowing for easy ingesting and extraction by Splunk.

Once the inputs and script are set up, restart the universal forwarder and data should begin showing up based on the scheduled time.

Then in Splunk, query the data as such:

index=win sourcetype=shellsweepplus
|  stats count by DetectionMethod ConfidenceScore suspiciousKeywords matchedHeuristics FilePath Entropy StDev Hash

Modify your query by DetectionMethod to see across the board how each differs across your files.

Run ShellSweepPlus

Similar to ShellSweep, it’s also possible to run ShellSweepPlus separately without Splunk. Maybe for incident response or validation other preventative controls may have missed. It’s as simple as a single command:

(ShellSweepPlus ASCII Art, Splunk 2024)

Closing Thoughts

As cybersecurity threats continue to evolve and become more sophisticated, it is important for organizations to adopt tools that can keep pace with these changes. ShellSweepPlus represents a step forward in web shell detection, offering a comprehensive and adaptable solution that can help organizations stay ahead of the curve.

We encourage readers to try ShellSweepPlus in their own environments and experience firsthand the benefits of its features.

Do you have ideas, suggestions, or questions? Your feedback is invaluable in helping us further refine and improve the tool to meet the ever-changing needs of the cybersecurity community.