Splunk Insights: Investigating the 3CXDesktopApp Supply Chain Compromise

CrowdStrike announced on 3/29/2023 that an active intrusion campaign was targeting 3CX customers utilizing a legitimate, signed binary, 3CXDesktopApp (CISA link). As the investigations and public information came out publicly from vendors all across the spectrum, C3X customers of all sizes began investigating their fleet for signs of compromise. These campaigns are often referred to as supply chain compromises, or MITRE ATT&CK T1195. The most notable of these attacks which brought supply chain security to the forefront of most organizations’ security posture was SolarWinds. A notable learning of dealing with the Solarwinds vulnerability was the difficulty associated with identifying supply chain compromises at the source. For the 3CXDesktopApp, it all began after a 7 day sleep that the compromised software version began to trigger different anti-virus products and showed suspicious behaviors in EDR products.

Organization defenders must consider attack surface comprising both endpoint and network. Utilizing our defense in depth approach, tracking anti-virus, EDR and other alerts provided can assist with piecing together the puzzle. It’s not a simple task when it comes to identifying software supply chain compromises. It may  all begin with a post-exploitation event and working backwards allows us to see the source.

In this Splunk blog post, we aim to equip defenders with the necessary tools and strategies to actively hunt down and counteract this campaign. Additionally, we will offer some resilient analytic ideas that can serve as a foundation for future threat detection and response efforts.

Infection Chain Walk Through

The supply chain compromise begins when users download an affected version of the 3CXDesktopApp, which subsequently loads a maliciously crafted or trojanized ffmpeg.dll. This compromised component is responsible for initiating the malicious activities associated with the attack.

Affected 3CX versions:

• 3CXDesktopApp-18.12.407.msi
• 3CXDesktopApp-18.12.416.msi

ffmpeg.dll 

The patched ffmpeg.dll is responsible for reading another DLL named "d3dcompiler_47.dll," which contains an encrypted shellcode and additional DLLs that will download several .ico files. Figure 1 presents a code snippet of the maliciously crafted ffmpeg.dll that reads the "d3dcompiler_47.dll" file to search for an embedded encrypted shellcode, starting with an 8-byte sequence "0xFE 0xED 0xFA 0xCE 0xFE 0xED 0xFA 0xCE."

^{Figure 1}

D3dcompiler_47.dll

The shellcode is encrypted using the RC4 algorithm, with a specific decryption key "3jB(2bsG#c7". Figure 2 illustrates the encrypted code block embedded in d3dcompiler_47.dll before and after the decryption process. Upon examining the decrypted portion of the screenshot, it becomes evident that the shellcode contains instructions to load another DLL.

^{Figure 2}

Decrypted-DLL

The shellcode proceeds to load the decrypted DLL export "DllGetClassObject," which initiates a thread to examine the manifest file. It then sleeps for a duration based on a randomly generated value relative to the system date and time. Following this, it reads the machine GUID from the registry. Figure 3 demonstrates how the shellcode accesses the Cryptography registry to parse the MachineGUID of the targeted or compromised host.

^{Figure 3}

Upon retrieving the Machine GUID, the shellcode calls a function that attempts to download several .ico files from the GitHub repository. At the time of writing, the URL link was no longer accessible, but the cybersecurity community shared the files, enabling us to examine the next stage.

Figure 4 presents a code snippet of the decrypted DLL that attempts to download multiple .ico files for decoding and decryption. The code highlights an intriguing approach employed by the attacker, using .ico files as configuration files. After downloading the .ico files, the shellcode reads them byte by byte, searching for the character "$". This character serves as a marker for the encoded and encrypted C2 URL link.

^{Figure 4}

Figure 5 presents a basic hex view snippet of two malicious .ico files that the decrypted DLL attempts to download. The hex bytes highlighted in the yellow box represent the base64-encoded and encrypted C2 URL link, which begins with the "$" character. We recommend using the decrypt-ico.py script created by the Volexity team to automatically decrypt this string. The decrypted C2 server can be found in the IOC section of this blog.

^{Figure 5}

The aforementioned C2 server proceeds to download an additional configuration JSON file, ultimately leading to the final payload binary, which is a browser stealer malware. This malware is designed to extract sensitive information from the victim's web browsers.

Browser Stealer Payload

The browser stealer is a separate x64-bit DLL that executes its malicious code through the "DllGetClassObject" export function. This malware aims to extract information such as domain name, computer name, and OS version using the NetWkstaGetInfo() and RtlGetVersion() APIs. Figure 6.1 and 6.2 display code snippets illustrating how the malware retrieves the specified information using these two Windows APIs and formats it before transmitting the data to its C2 server.

^{Figure 6.1} 

^{Figure 6.2}

Finally, the malware targets several well-known browsers, including "Chrome," "Firefox," "MSEdge," and "Brave," in order to steal information. It achieves this by accessing browser history and the places.sqlite database, copying it, and then querying the discovered SQLite browser databases to parse the URL and title, limited to the first 500 entries. Figure 7 displays a code snippet illustrating how the stealer executes the SQL command once it locates the browser SQLite database it needs to parse and subsequently sends the information to its C2 server.

^{Figure 7}

IOCs

We identified several key factors during our analysis that aid in guiding Splunk content creation. Now, let's delve into the content and examine the various ways in which Splunk can be of assistance.

Security Content

There are numerous methods for generating content in Splunk, as well as a wide variety of data sources. Based on the indicators provided and our analysis above, we can present the following content. Some of these examples may serve as Splunk inspiration, while others may be suitable for notables. Throughout our discussion, we will offer insights on building resilient analytics for each example.

Hunting 3CXDesktopApp Software

Initially, like many, we want to identify endpoints across our fleet that have C3XdesktopApp running and what version. We decided to use the Endpoint.Processes datamodel so the results would be back fast. If data is not normalized in the datamodel, that’s ok! Modify the analytic for your environment by looking for the process names. Note here that the datamodel does not provide file version, we are specifically just looking for where this process is running across the fleet.

| tstats `security_content_summariesonly` count min(_time) as firstTime max(_time) as lastTime from datamodel=Endpoint.Processes where Processes.process_name=3CXDesktopApp.exe OR Processes.process_name="3CX Desktop App" by Processes.dest Processes.user Processes.parent_process_name Processes.process_name Processes.original_file_name Processes.process Processes.process_id Processes.parent_process_id 
| `drop_dm_object_name(Processes)` 
| `security_content_ctime(firstTime)` 
| `security_content_ctime(lastTime)`

Two aspects we recommend examining closely at this time are the file path and the command line. These elements may vary across different environments, so it's important to identify the default location of the binary for your organization and determine if the command line follows a consistent pattern.

| tstats `security_content_summariesonly` count min(_time) as firstTime max(_time) as lastTime from datamodel=Endpoint.Processes where Processes.process_name=3CXDesktopApp.exe OR Processes.process_name="3CX Desktop App" by Processes.process_path Processes.process_name 
| `drop_dm_object_name(Processes)` 
| `security_content_ctime(firstTime)` 
| `security_content_ctime(lastTime)`

Windows Vulnerable 3CX Software

Switching to Sysmon, we wrote a query to look for 3CXDesktopApp by file version. Depending on the EDR product in use, many provide signature information, VirusTotal enrichment, prevalence and so forth. 

The Splunk Attack Range uses a broad configuration file meant to capture every artifact provided. Each EDR product today provides similar or more, so it is very important to understand the product and how it can assist your organization in an event like this.

`sysmon` (process_name=3CXDesktopApp.exe OR OriginalFileName=3CXDesktopApp.exe)  FileVersion=18.12.* 
| rename Computer as dest 
| stats count min(_time) as firstTime max(_time) as lastTime by dest, parent_process_name,process_name, OriginalFileName, CommandLine 

According to 3CX, the security issue affects version numbers 18.12.407 and 18.12.416 on Windows. We adopt a slightly broader approach by searching for any 18.12.* version, primarily to monitor for any instances that may have gone unnoticed. Furthermore, you can modify this analytic to examine any version or simply extract the version information for an inventory overview.

Another take on this query showing just the process and version number by host. 

`sysmon` (process_name=3CXDesktopApp.exe OR OriginalFileName=3CXDesktopApp.exe)
| rename Computer as dest 
| stats count min(_time) as firstTime max(_time) as lastTime by dest process_name FileVersion

18.12.422 is the latest version as of 3/31/2023.

3CX Supply Chain Attack Network Indicators

We would like to thank CrowdStrike and numerous other organizations for providing indicators. The method for detecting the domains used will depend on an organization's security stack. Some products reveal the URI, while others do not. In our case, we utilize DNS queries from Sysmon, which populates the Network_Resolution data model.

Hunting with these domains may provide false positives and filtering / tuning is definitely recommended. Note here that a hit on the domain is not 100% true positive. Some of these are legitimate and will require further review. In addition to looking for the domains, it may provide value in doing two additional tasks based on product support:

1. Restrict the network indicators to 3CXDesktopApp, or broadly any process
2. Add URIs to the lookup, or a new query, and hunt for beaconing activity.

| tstats `summariesonly` values(DNS.answer) as IPs min(_time) as firstTime from datamodel=Network_Resolution by DNS.src, DNS.query 
| `drop_dm_object_name(DNS)` 
| `security_content_ctime(firstTime)` 
| `security_content_ctime(lastTime)` 
| lookup 3cx_ioc_domains domain as query OUTPUT Description isIOC 
| search isIOC=true

Utilizing the Splunk App for Lookup File Editing, we can easily add/remove indicators or new columns.

DLLs on Disk

As mentioned earlier, it is important to pay attention to the process path. In this specific campaign, we aim to identify any additional files that were dropped on the disk, collect their hashes, and explore potential leads that may offer further insights. Using Sysmon, we have narrowed our focus to the \Appdata\local\ path and sorted the data by the ImageLoaded (DLL) and various metadata points that Sysmon offers. It's important to note that different EDR products will provide varying levels of visibility, so as you analyze this telemetry, start identifying alternative ways to pivot. Be sure to check for prevalence within your organization. For example, if the ffmpeg.dll with this specific hash is found on only 5 out of 5,000 endpoints, it is certainly worth investigating further.

`sysmon` 3cxdesktopapp.exe  ImageLoaded="C:\\Users\\Administrator\\AppData\\Local\\Programs\\3CXDesktopApp\\*" | stats values(ImageLoaded) by loaded_file MD5 FileVersion Company Description service_dll_signature_verified

#ToolTips

Image loads are a voluminous datasource and can be cumbersome to hunt through. Here are some tips to narrow down interesting image loads.

1. Focus on non-standard paths. Native Windows DLLs will not run out of the user profile
2. Identify signing information and use it to your advantage to look for Unsigned or revoked based on file paths
3. If possible, look for processes loading DLLs from non-standard paths. Filter by signing status. 

Registry

Revisiting the initial installation process involving MsiExec.exe, it's important to note that several registry modifications occur to ensure the persistence of this version of 3CXDesktopApp. 

`sysmon` EventID IN (12,13,14)   process_name="msiexec.exe" *\\appdata\\* 
| stats values(registry_value_data) by registry_path

Now the registry modifications from the 3CXDesktopApp. This is an abbreviated version as there are a lot of standard modifications in the output.

`sysmon` EventID IN (12,13,14)   process_name="3cxdesktopapp.exe" 
| stats values(registry_value_data) by registry_path

Learn More

You can find the latest content and security analytic stories on GitHub and in Splunkbase. Splunk Security Essentials also has all these detections available via push update. 

For a full list of security content, check out the release notes on Splunk Docs.

Feedback

Any feedback or requests? Feel free to put in an issue on GitHub and we’ll follow up. Alternatively, join us on the Slack channel #security-research. Follow these instructions If you need an invitation to our Splunk user groups on Slack.

Contributors

We would like to thank Michael Haag  and Teoderick Contreras for authoring this post and the entire Splunk Threat Research Team (Rod Soto, Mauricio Velazco, Lou Stella, Bhavin Patel, Eric McGinnis, and Patrick Bareiss) for their contribution to this release.

References:

• https://www.3cx.com/blog/news/desktopapp-security-alert/
• https://www.volexity.com/blog/2023/03/30/3cx-supply-chain-compromise-leads-to-iconic-incident/
• https://www.sentinelone.com/blog/smoothoperator-ongoing-campaign-trojanizes-3cx-software-in-software-supply-chain-attack/
• https://www.reddit.com/r/crowdstrike/comments/125r3uu/20230329_situational_awareness_crowdstrike/
• https://www.cisa.gov/news-events/alerts/2023/03/30/supply-chain-attack-against-3cxdesktopapp
• https://www.3cx.com/community/threads/crowdstrike-endpoint-security-detection-re-3cx-desktop-app.119934/page-2#post-558898
• https://www.3cx.com/community/threads/3cx-desktopapp-security-alert.119951/