Monitoring Pulse Connect Secure With Splunk (CISA Emergency Directive 21-03)

Splunk is committed to using inclusive and unbiased language. This blog post might contain terminology that we no longer use. For more information on our updated terminology and our stance on biased language, please visit our blog post. We appreciate your understanding as we work towards making our community more inclusive for everyone.

Contributors: Mick Baccio, James Brodsky, Tamara Chacon, Shannon Davis, Dave Herrald, Kelly Huang, Ryan Kovar, Marcus LaFerrerra, Michael Natkin, John Stoner and Bill Wright

Update May 4, 2021: Over the last two weeks, there have been several significant developments. First and most importantly, Pulse Secure issued an update on May 3 addressing multiple vulnerabilities. Splunk recommends that all Pulse Secure users review and install the update as soon as possible. On April 30, CISA updated Alert (AA21-110A) with new detections, including the "Impossible Travel" detection and JA3 analysis. We have updated our Splunk-friendly collection of indicators to include the latest from CISA.

To immediately see how to find potential vulnerabilities or exploits in your Pulse Connect Secure appliance, skip down to the "Identifying, Monitoring and Hunting with Splunk" section. Otherwise, read on for a quick breakdown of what happened, how to detect it, and MITRE ATT&CK mappings.

What You Need to Know About the Pulse Connect Secure Attacks

Over the past few weeks, there has been increasing chatter regarding adversary groups exploiting multiple vulnerabilities in the Pulse Connect Secure (PCS) virtual private network (VPN) appliance. On April 20, 2021, the Mandiant team at FireEye released a blog detailing their findings from multiple recent incidents involving compromised PCS appliances. This report prompted a flurry of activity from various organizations, including government agencies and security vendors. That same day, DHS Cybersecurity and Infrastructure Security Agency (CISA) released Alert (AA21-110A) and Emergency Directive 21-03, the latter requiring all US Federal agencies to take specific action concerning PCS appliances in their environments. Splunk recommends all US Federal agencies refer to the DHS directive to ensure compliance.

According to a blog post by Pulse Secure, the incidents disclosed this week involve vulnerabilities that were patched in 2019 and 2020, plus a new issue (CVE-2021-22893 Security Advisory SA44784) discovered this month. The vendor notes that a software update for this new issue will be available in early May. The post contains valuable information on all the vulnerabilities and recommended mitigation measures, and customer support information. Of particular importance is the Pulse Connect Secure Integrity Tool, which allows you to check if essential components of your PCS appliance software have been tampered with. Splunk recommends all PCS customers follow this vendor-published guidance in its entirety.

The PCS appliance is a popular VPN solution that offers workers secure access to an organization's internal networks from anywhere in the world. Because VPN appliances play a critical role in securing an organization’s network perimeter and, by design, exposed directly to the Internet, they are often targeted by adversaries. Although the latest news is specific to Pulse Secure products, attacks of this nature are not limited to one vendor. Adversaries commonly return to vulnerabilities in VPN products to gain unauthorized access to organizations. As recently as April 16, 2021, the US National Security Agency released a cybersecurity advisory warning that older vulnerabilities in at least five different remote access products were being actively exploited.

Identifying, Monitoring, and Hunting with Splunk

Here are some hot-off-the-press searches to help find some of the badness described in the FireEye/Mandiant blog and other sources. If we have coverage for these searches in Splunk Security Content, we call them out in the MITRE ATT&CK section.

Note that proof of concept (POC) exploits are only available for some of the vulnerabilities involved in this attack. The detections below are all derived from a lab environment and informed by the context provided in the FireEye/Mandiant report. If we uncover more information, we will publish updates!

Ingesting PCS Appliance Data into Splunk

Several of the following techniques require logs from your PCS appliances to be ingested into Splunk for analysis. PCS appliances use syslog to log to external systems like Splunk. Ivanti publishes configuration details in the PCS appliance Admin guide; here is a link to Admin Guide version 9.1R11 (the latest available at the time of publication). PCS appliances support multiple log export formats ("standard," "WELF," "W3C"). Each of these formats yields slightly different data and results. In 2015, Juniper Networks divested the JunOS Pulse product line to Pulse Secure, LLC, now owned by Ivanti. Early versions of the Splunk Add-on for Juniper included support for the IVE/SA (now PCS) product line, with a sourcetype of "juniper_sslvpn" or "juniper:ive." Since this time, third parties have published their own TAs on SplunkBase (such as this one) or GitHub (such as this one) to continue to support the PCS appliance log data. BOTS V’s RWI scenario made use of WELF-formatted log data from a Juniper SA utilizing this TA. Note that some examples in this blog use a sourcetype of ‘juniper:ive,’ some use a sourcetype of ‘pulse:connectsecure’, and some use a sourcetype of ‘syslog.’ This emulates the variety of formatting we’ve observed in the wild. You should determine the sourcetype in use in your environment and substitute it in these example searches when necessary.

Additionally, we recommend enabling logging for "Unauthenticated Requests" (this setting is disabled by default on PCS appliances). While this may slightly increase the log volume, the CISA alert demonstrates that it is vital in revealing evidence of tampering and potentially ongoing malevolent activity.

Version Checking

According to this week’s advisories, certain versions of the Pulse Secure appliance software contain critical vulnerabilities. Therefore, it is important to catalog the versions of all Pulse Secure Connect appliances you have on your network so that you can create a mitigation plan. You can use Splunk to do this if you don’t have an alternative method. If you already have a list of IP addresses for your PCS appliances, you can feed that list as a lookup into a simple Splunk Technical Add-On that we have created, which will sequentially probe each appliance in the list and report the PCS version found into Splunk for reporting or alerting.

index=main source="checkpcsversions.sh"
| stats values(pcs_version) as "PCS Version" by dest_ip
| iplocation dest_ip

CVE-2021-22893 Mitigation Monitoring

This technique can be employed to assess your exposure by finding appliances that may still require mitigation according to the Pulse Secure guidance. We suggest you search through those logs for evidence of the following:

• Windows File Share Browse
• Pulse Secure Collaboration 
  
  

Why? Because both of those services are vulnerable, and the temporary "workaround" mitigation that you should have already applied turns both of these functions off. Therefore, if you are still seeing activity from either function in the logs, and you know that your appliance is running a vulnerable Pulse Secure version, you must apply the appropriate workaround via application of the "workaround-2104.xml" file. You can create a report or alerts based on any appliances still logging file sharing or collaboration activity, and provide the report or alerts to your network team to implement the workaround. Pulse Secure has committed to releasing an update in early May 2021, at which time this workaround can be reverted, thereby restoring these two capabilities if necessary. 

Pulse Secure Collaboration Activity

Here is a quick and easy search to determine if Pulse Secure Collaboration is currently being used. You might look at the search and think, what is this reference to "meetings?" Well, "Secure Meetings" is what the original functionality in the appliance was called. Not all of the meeting fields are displayed here nor are they all required to determine if it is running, but it is meant to show you what can be seen in the events.

sourcetype=pulse:connectsecure dest=<IP Address of Pulse devices> meeting_id=*

| table _time src meeting_name meeting_action meeting_bytes_read meeting_bytes_written meeting_client_version meeting_invitee meeting_server meeting_property meeting_property_initial_value meeting_property_new_value meeting_atendee meeting_duration

| sort + _time

Note: You should determine the sourcetype in use in your environment and substitute it in these example searches as applicable.

Windows File Share Activity

As stated, you’re also going to want to track Windows File Share activity. If your PCS appliance is configured to use WebTrends Enhanced Log Format (WELF) formatting for syslog events, this activity will be captured in the proto field. Search your logs for: 

proto=fbr for Windows File Share activity

And as an alternative to the meeting search above, you can also search:

proto=meeting for Pulse Secure Collaboration activity

An example of this, combining both kinds of activity in the same search, is as follows:

search sourcetype=juniper:ive proto!=auth
| stats values(proto) AS Activity by user

Note: You should determine the sourcetype in use in your environment and substitute it in these example searches as applicable.

However, if your PCS appliance is configured for "standard" formatting for your syslog, PCS may identify activity based on a message code within the "msg" field. The format is three letters followed by five numbers (e.g. "MTG20035" or "FBR20503"). The examples below extract only the three letter code which indicates the feature being used. As previously noted, FBR denotes Windows File Share activity.

Examples of this are as follows:

sourcetype="juniper:ive"
| rex field=msg "(?<product>^\w{3})"
| stats values(product) AS Activity by user

In the following example, we’re substituting the activity code in-line (though this could be accomplished with a lookup as well):

sourcetype="juniper:ive"
| rex field=msg "(?<product>^\w{3})"
| eval app=case(match(product,"FBR"),"Windows File Browser",match(product,"JAV"),"Java Embedded App",match(product,"MTG"),"Secure Meeting")
| stats values(app) AS Activity by user

Note: You should determine the sourcetype in use in your environment and substitute it in these example searches as applicable.

Alternatively, if your logging includes URIs, you may search for the following:

/dana/fb/smb/wfb.cgi for Windows File Share browsing activity

/dana/fb/smb/swsh.cgi   for Windows File Share activity

/dana-na/{meeting_base_URL}/meetingrun.cgi for Pulse Secure Collaboration activity (where {meeting_base_URL} is the administrator-configured base URL for secure meetings)

Depending on the logging configuration on the PCS appliance, events such as the ones below are logged when connections are attempted to access files on the Windows servers.

Turning to a familiar source of data that you may also be collecting, Windows Security event logs, the search below provides a way to gain insight into files being accessed through a share directly from Windows. This can be useful in case the PCS appliance is not logging this level of fidelity.

source="WinEventLog:Security" EventCode=5145 Source_Address=<IP Address of Pulse devices> Share_Name="\\\\*\\C$"

| table _time ComputerName Security_ID Relative_Target_Name Accesses

Similarly, if a share is attempted to be accessed without proper permissions, a Windows EventCode 4625 will trigger. Turning back to the Pulse Secure logs, in the example below, the user "badguy" successfully joins as shown in these events:

However, “badguy” tries to connect to a file share and is rejected. The following search sheds light on why he was not able to access the desired resource. In this example with LDAP configured on the Windows device, the "user name does not exist." This search can be used to determine why login events from that originate from the Pulse Server(s) are not authenticating successfully.

source="WinEventLog:Security" EventCode=4625  Source_Network_Address=<IP Address of Pulse devices>  

| table  _time EventCode hostname user Source_Network_Address Logon_Process

Why is all of that important? Because, it enables another potential detection. FireEye noted in their blog that “the malicious operations of SLOWPULSE can be detected via log correlation between the authentication servers responsible for LDAP and RADIUS auth and the VPN server. Authentication failures in either LDAP or RADIUS logs with the associated VPN logins showing success would be an anomalous event worthy of flagging.” The following search will return a listing of login events from Pulse that are tied into LDAP; in our case we are using Active Directory for LDAP so our realm is called AD.

sourcetype=pulse:connectsecure dest=<IP Address of Pulse devices> action=success realm=AD
| table _time user
| sort + _time

From there, we could take those Pulse logins and look for the failed login events, Event Code 4625. That search is above. Timestamps should be very close to one another, within seconds.

Note: You should determine the sourcetype in use in your environment and substitute it in these example searches as applicable.

Indicators of Compromise (IOCs)

FireEye/Mandiant published IOCs, including files, hashes, web shells, and sed patterns in their blog posts. Ivanti and CISA published additional IOCs (including additional hashes and affected URIs) in their respective posts. We have consolidated and converted these indicators into simple CSV format so that you may use them as lookup tables — and posted to Github here for review and consumption. But what’s a lookup table, and how does it help with security detection in Splunk? Got you covered there, too.

Webshells

Many of the attacks disclosed this week result in the installation of webshells on the PCS appliance. We felt it valuable to include a reference to this classic Youtube video from James Bower about how to hunt web shells with Splunk.

Splunk Enterprise Security

Know Thyself

While we have spent some time explaining this attack and effort needs to be put toward investigating this, it is also important to note that the basics are important. Basic asset management, hopefully via your asset and identity framework, will tell you where your vulnerable systems reside. Running regular vulnerability scans that integrate into Splunk will display which systems are vulnerable and can help you prioritize your patching schedule and better focus your detection efforts.

Threat Intelligence Framework

You can also leverage the IOCs above (for those that don’t want to scroll up, here they are) with Splunk Enterprise Security. The Threat Intelligence Framework can easily ingest these IOCs and Splunk Enterprise Security will let you know if any of them are identified. If you aren’t sure how to do this, check out our recent blog post that walks you through the process of onboarding IOCs in Splunk Enterprise Security.

MITRE ATT&CK

Reviewing the blog posts from FireEye/Mandiant, we mapped the adversary’s activity to MITRE ATT&CK. Each tactic is then linked to Splunk Content to help you hunt for that information. Be aware; these searches are provided as a way to accelerate your hunting. We recommend you configure them via the Splunk Security Essentials App. You may need to modify them to work in your environment! Many of these searches are optimized for use with the tstats command.

Finally, as more information becomes available, we will update these searches if more ATT&CK TTPs become known.

Here is the full list of all the MITRE ATT&CK TTP’s that FireEye/Mandiant reported as being associated with malware/groups identified in this attack: 

T1003,T1016, T1021.001, T1027, T1036.005, T1048, T1049, T1053, T1057, T1059, T1059.003, T1070, T1070.001, T1070.004, T1071.001, T1082, T1098, T1105,T1111,T1133,T1134.001, T1136,T1140,T1190,T1505.003,T1518,T1554,T1556.004,T1592.004, T1562,T1569.002,T1574, T1600

Conclusion

We know that these are significant vulnerabilities and that customers will want to patch as soon as possible and determine if they were affected by this attack. If you haven’t patched yet (we’ve all been there), hopefully, these searches will provide you the ability to have more visibility into your environment and any malicious activity that you might be experiencing. If they don’t work perfectly, think of them as "SplunkSpiration" :-). As soon as we have more information, we will update this blog. Need Help? Splunk experts are here for you! Customers should reach out to their customer success manager, sales engineer, or regional sales manager.

----------------------------------------------------
Thanks!
Dave Herrald