Splunk Security Advisory for Apache Log4j (CVE-2021-44228, CVE-2021-45046 and others)

Splunk is committed to using inclusive and unbiased language. This blog post might contain terminology that we no longer use. For more information on our updated terminology and our stance on biased language, please visit our blog post. We appreciate your understanding as we work towards making our community more inclusive for everyone.

Updated 8:30 am PT, 1/7/22

On December 10, a critical remote code execution vulnerability impacting at least Apache Log4j 2 (versions 2.0 to 2.14.1) was announced by Apache. This vulnerability is designated by Mitre as CVE-2021-44228 with the highest severity rating of 10.0. The vulnerability is also known as Log4Shell by security researchers. Log4j 2 is a commonly used open source third party Java logging library used in software applications and services. If exploited, this vulnerability allows adversaries to potentially take full control of the impacted system.

On December 14, Apache announced a second vulnerability impacting Log4j (CVE-2021-45046), found in Log4j version 2.1.0. On December 17, this vulnerability was upgraded by MITRE to a severity rating of 9.0 (Critical).

Splunk is focused on the fastest possible remediations for CVE-2021-44228 and CVE-2021-45046. Release candidates to address both vulnerabilities are in development for affected products, inclusive of the products listed below. Please return to this posting for the most up to date information. 

Splunk is currently reviewing our supported products for impact and evaluating options for remediation and/or or mitigation. This includes implementing additional proactive measures within Splunk's internal environment and Splunkbase to address the dynamic threats related to CVE-2021-44228 and CVE-2021-45046. The below tables contain our most up-to-date guidance on our products. These products are tracked separately across On Prem and Cloud products.

Splunk has not observed successful exploitation of the Log4Shell vulnerability within Splunk Cloud. Splunk has also not observed successful exploitation of the Log4Shell vulnerability within our internal environment. Splunk does not have visibility into On-Prem deployments. Please see our blogs for guidance on detecting and protecting your deployment from Log4Shell: 

• Detecting Log4j 2 RCE Using Splunk
• Detecting Log4j Vulnerability Continued
• Simulating, Detecting, and Responding to Log4Shell with Splunk

Please return to this posting for the most up to date information. Current customers can file support tickets through standard channels for specific guidance.

Supplemental Security Advisory for Splunk Apps

A supplemental security advisory for Splunk Apps was published on December 14 and is being updated on an ongoing basis. 

Additional Guidance for CVE-2021-45105 and CVE-2021-44832

Splunk also reviewed a Denial of Service Vulnerability (CVE-2021-45105) found in Log4j version 2.16.0. Apache has designated this vulnerability a severity rating of 7.5 (High). Per Apache’s advisory, specific non-default configuration parameters need to be present to exploit this vulnerability. Splunk has evaluated where these configuration parameters may exist within our product portfolio, and we have updated the table below accordingly.

Splunk is additionally reviewing a Remote Code Execution Vulnerability (CVE-2021-44832) found in Log4j version 2.17.0. Apache has designated this vulnerability a severity rating of 6.6 (Moderate). Per Apache’s advisory, permission must be granted to the underlying configuration files, and a malicious configuration needs to be created, to exploit this vulnerability. 

Unless CVE-2021-45105 or CVE-2021-44832 increase in severity, Splunk will address these vulnerabilities as part of the next regular maintenance release of each affected product. Customers also have the option to remove Log4j Version 2 from Splunk Enterprise out of an abundance of caution. 

Summary of Impact for Splunk Enterprise and Splunk Cloud

Core Splunk Enterprise functionality does not use Log4j version 2 and is not impacted. If Data Fabric Search (DFS) is used, there is an impact because this product feature leverages Log4j. If this feature is not used, there is no active attack vector related to CVE-2021-44228 or CVE-2021-45046. Guidance for determining if you are using DFS appears in the "Removing Log4j version 2 from Splunk Enterprise" section below. 

All recent non-Windows versions of Splunk Enterprise include Log4j version 2 for the DFS feature. Windows versions of Splunk Enterprise do not include Log4j version 2. Customers may follow the guidance in the “Removing Log4j version 2 from Splunk Enterprise” section below to remove these packages out of an abundance of caution. Official patches to upgrade the Log4j packages and mitigate the vulnerabilities in all usage scenarios are available and linked in the table below for version 8.1 and 8.2. These patches are the preferred method for addressing CVE-2021-44228 in Splunk Enterprise. Patches to address CVE-2021-45046 are forthcoming.  

Splunk Cloud is not impacted by CVE-2021-44228 or CVE-2021-45046. For potential impact on Splunk supported applications installed on Splunk Enterprise or Splunk Cloud, see the tables below.

Impacted Products

These products are known to be impacted by CVE-2021-44228 and CVE-2021-45046. Unless explicitly stated, patches are cumulative to address both CVE-2021-44228 and CVE-2021-45046. The latest available update for an affected product should be used. 

Products Confirmed Not Vulnerable

Investigation has concluded that these products are not impacted by CVE-2021-44228 or CVE-2021-45046.

• Admin Config Service
• Analytics Workspace
• Behavior Analytics
• Dashboard Studio
• Developer Tools: AppInspect
• Enterprise Security
• Infosec App for Splunk
• Intelligence Management (TruSTAR)
• KV Service
• Mission Control
• MLTK
• Operator for Kubernetes
• Security Analytics for AWS
• SignalFx Smart Agent
• SOAR Cloud (Phantom)
• SOAR (On-Premises)
• SOAR Cloud On-Prem Automation Broker
• Splunk Augmented Reality
• Splunk Cloud Data Manager (SCDM)
• Splunk Connect for Kubernetes
• Splunk Connect for SNMP
• Splunk Connect for Syslog
• Splunk DB Connect
• Splunk Enterprise Cloud
• Splunk Log Observer
• Splunk Mint
• Splunk Mobile
• Splunk Network Performance Monitoring
• Splunk Open Telemetry Distributions
• Splunk Profiling
• Splunk Secure Gateway (Spacebridge)
• Splunk Security Essentials
• Splunk TV
• Splunk Universal Forwarder (UF)
• Splunk User Behavior Analytics (UBA)
• Stream Processor Service

Removing Log4j Version 2 from Splunk Enterprise

The guidance in this section is intended to be used in the case that Splunk Enterprise cannot be upgraded using the official patches for version 8.1 and 8.2. The guidance below will help you remove jar files associated with both vulnerabilities (CVE-2021-42288 and CVE-2021-45046).

If the Splunk Enterprise instance does not leverage DFS, the presence of those libraries does not introduce an active attack vector. Out of an abundance of caution, you may remove the unused jar files and directories from your Splunk Enterprise instances in the following paths:

• $SPLUNK_HOME/bin/jars/vendors/spark
• $SPLUNK_HOME/bin/jars/vendors/libs/splunk-library-javalogging-*.jar
• $SPLUNK_HOME/bin/jars/thirdparty/hive*
• $SPLUNK_HOME/etc/apps/splunk_archiver/java-bin/jars/*

Upon removal of these jar files, an administrator may see errors at Splunk startup pertaining to file integrity, specific to these jar files. These are expected as you are removing these unused jar files as a workaround. These errors may be ignored. 

Jar files matching the same filename of the files found in the directories above, but found in other directories on your Splunk instances are likely from normal Splunk operation (e.g. search head bundle replication) and can be safely deleted. If any jar files return in the splunk_archiver app, disabling the default Bucket Copy Trigger search in that app will stop this behavior from happening. 

*Since a Splunk Heavyweight Forwarder (HWF) is a full-instance copy of Splunk Enterprise with forwarding enabled, the above mitigation may also be applied to HWF instances.

Determining if DFS is in use

To determine if Distributed Fabric Search is in use, you may run the following query from a Splunk search head:

| history 
| search search=*dfsjob* 
|  rex field=search "(?P<dfs_cmd>\|\s*dfsjob)" 
| search dfs_cmd=* and search!=*eval* 
| where len(dfs_cmd) > 0

If the above search returns results, then DFS is enabled and searches have been run using the capability. You may also look for the parameter "disabled=false" in server.conf to determine if DFS is enabled. 

Determining if Hadoop Data Roll is in use

Although Hadoop Data Roll (archiver) functionality does not introduce an active attack vector, users who do not use this functionality may choose to remove the Log4j files out of an abundance of caution. To determine if this feature is in use, you may run the following query from a Splunk search head:

index=_internal source=*/splunk_archiver.log
| rex field=_raw "json=\"(?P<json>.*)\"" 
| chart values(json)

If the above search returns the following, then Hadoop Data Roll is NOT in use:

Unsupported Versions of Splunk Enterprise

Only the DFS functionality of unsupported versions of Splunk Enterprise that include DFS (the 8.0 release and later) is affected by CVE-2021-44228 and CVE-2021-45046. The above removal guidance can be applied to those versions as well. Splunk has provided an official patch for supported versions 8.1.7.1 and 8.2.3.2. 

Removing Log4j Version 2 from Splunk User Behavior Analytics

Versions of UBA prior to 5.0 leveraged Apache Storm, which embeds Log4j. The presence of those libraries does not introduce an active attack vector. Out of an abundance of caution, you may follow the procedure here to completely remove Apache Storm and the Log4j libraries from your UBA AMI, OVA, and bare-metal installs.

References

• https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-44228
• https://logging.apache.org/log4j/2.x/security.html
• https://www.splunk.com/en_us/cyber-security/log4shell-log4j-response-overview.html
• https://www.splunk.com/en_us/blog/security/log-jammin-log4j-2-rce.html
• https://www.splunk.com/en_us/blog/security/log4shell-detecting-log4j-vulnerability-cve-2021-44228-continued.html
• https://www.splunk.com/en_us/blog/security/simulating-detecting-and-responding-to-log4shell-with-splunk.html
• https://www.splunk.com/en_us/blog/bulletins/supplementary-security-advisory-for-splunk-apps-add-ons.html
• https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-45046 
• https://nvd.nist.gov/vuln/detail/CVE-2021-45105
• https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-44832 

Change Log 

• 2022-01-06: Updated advisory to include instructions on removing Apache Storm from older versions of Splunk User Behavior Analytics.
• 2021-12-30: Updated advisory to acknowledge the multiple vulnerabilities that have been identified since December 10. Added CVE-2021-44832 MITRE designation in References section.
• 2021-12-23: Updated Splunk Enterprise - CVE-2021-45046: 8.1.7.2, 8.2.3.3 with the addition of 8.2.4.
• 2021-12-21: Updated fixed versions for Splunk Enterprise Amazon AMI for CVE-2021-40546. Updated fixed versions for Data Stream Processor. Added fix information for CVE-2021-40546 for the following products: Splunk On-call / VictorOps; Splunk Real User Monitoring; Splunk Application Performance Monitoring; Splunk Infrastructure Monitoring; Splunk Log Observer; Splunk Synthetics. Added fixed version for Splunk Connect for Kafka for CVE-2021-45105. Added SOAR Cloud On-Prem Automation Broker to list of products confirmed not vulnerable
• 2021-12-20: Updated fixed versions of Splunk Enterprise Docker Container for CVE-2021-44228 and CVE-2021-45046. Updated list of products not vulnerable to CVE-2021-45105.
• 2021-12-18: Updated advisory to reflect Splunk Enterprise, IT Service Intelligence, IT Essentials Work, and Data Stream Processor are not vulnerable to CVE-2021-45105
• 2021-12-18: Added additional guidance for CVE-2021-45105
• 2021-12-18: Added fix versions for Splunk Enterprise, Splunk Enterprise AMI, and Splunk Enterprise Docker images addressing CVE-2021-45046
• 2021-12-17: Updated advisory to reflect the MITRE-upgraded severity rating of CVE-2021-45046 to 9.0 (Critical). Updated Splunk’s combined approach to vulnerabilities CVE-2021-44228 and CVE-2021-45046. Added link to new Splunk blog - Simulating, Detecting and Responding to Log4Shell with Splunk.
• 2021-12-17: Updated advisory with additional products confirmed vulnerable: Splunk VMWare OVA for ITSI and Splunk UBA OVA Software.  Added additional products confirmed not vulnerable: Infosec App for Splunk and Splunk Security Essentials.  Added fixed version 8.1.7.1 for Splunk Enterprise AMI
• 2021-12-16: Moved Stream Processor Service from Impacted to Not Vulnerable list.  Note that while SPS was patched on 12/16 to protect against CVE-2021-44228, this service was never vulnerable to either CVE-2021-44228 or CVE-2021-45026 due to the specific service implementation.
• 2021-12-16:  Clarified that no workarounds will be published for versions already patched.  Updated Splunk Application Performance Monitoring, Splunk Infrastructure Monitoring, Splunk On-Call/VictorOps, Splunk Real User Monitoring, and Splunk Synthetics to note a previous (now patched) vulnerability to CVE-2021-44228, and pending patches for CVE-2021-45046.
• 2021-12-16: Updated with additional products confirmed vulnerable: Splunk OVA for VMWare and Splunk OVA for VMWare Metrics. Added link to Splunk.com Log4Shell information hub in References section
• 2021-12-16: Added fix versions for Stream Processor Service and Splunk Logging Library for Java
• 2021-12-15: Clarified the status of Splunk deployments within our corporate or customer’s environments with regard to Log4Shell. Added App IDs to impacted products.
• 2021-12-15: Added fix versions for ITSI and IT Essentials Work. Updated impacted versions of Splunk Logging Library for Java
• 2021-12-14: Added link to supplemental security advisory for Splunk Apps and updated fixed versions of IT Essentials Work. Linked to Splunk docs in workaround column for IT Essentials and ITSI
• 2021-12:14: Added guidance for CVE-2021-45046. Updated impacted versions of Splunk Connect for Kafka. Added links to second Log4Shell Splunk blog post and CVE-2021-45046 MITRE designation in References section
• 2021-12-14: Clarified official names of impacted add-ons. Added fix version for Splunk Enterprise AWS AMI, Splunk Add-on for JBoss, Splunk Add-on for Tomcat and Splunk Add-on for Java Management Extensions. Added additional fix versions for ITSI and Splunk Essentials Work
• 2021-12-13: Added link to patch for Splunk Enterprise 8.2.3.2 and additional information about mitigating vulnerabilities in earlier Splunk Enterprise versions by removing Log4j jar files
• 2021-12-13: Added fix version 4.9.5 for ITSI and IT Essentials Work. Added link to patch for Splunk Enterprise 8.1.7.1
• 2021-12-13: Added fix version 4.10.3 as available for ITSI and IT Essentials Work. Corrected impacted version numbers for Java Management Extensions Add-on
• 2021-12-13: Updated advisory to remove Hadoop (Hunk) integration as a risk vector for Splunk Enterprise. Added fix version 4.11.1 as available for ITSI. Confirmed vulnerability in product IT Essentials Work and added fix version 4.11.1
• 2021-12-12: Updated advisory with additional products confirmed not vulnerable including Splunk SDKs. Confirmed vulnerability in product Splunk Logging Library for Java. Updated timeline for fixed version available for ITSI. Added 8.2.3.2 in the expected release of Splunk Enterprise
• 2021-12-12: Removed advisory for DB Connect (was never impacted). Added official product names for UF, UBA, Phantom (On-Premises), HWF. Updated advisory with additional products confirmed not vulnerable including Splunk Connect for Kubernetes
• 2021-12-12: Updated advisory with additional products confirmed not vulnerable including Splunk Mint, Splunk Connect for SNMP, SignalFX Smart Agent and Splunk Forwarders (UF/HWF). Confirmed vulnerability in products Splunk DB Connect, Splunk Connect for Kafka, Add-On: Tomcat, Add-On: Java Management Extension and Add-On: JBoss
• 2021-12-11: Updated advisory with additional products confirmed not vulnerable including Admin Config Service, Behavioral Analytics, Data Manager, Enterprise Security, Intelligence Management (TruSTAR), KV Service, Mission Control, Phantom (On Premises), Security Analytics for AWS, SOAR Cloud (Phantom), Splunk Connect for Syslog, Splunk Mobile, Splunk OpenTelemetry Distributions, Splunk Operator for Kubernetes, Splunk Secure Gateway (Spacebridge) and Splunk TV
• 2021-12-11: Initial Security Advisory