Log4Shell, Splunkbase, and You: A Message From Your Friendly Neighborhood Cybersecurity Engineer

Call me David. As you might have heard, Log4Shell, “the single biggest, most critical vulnerability ever”¹ was recently disclosed to the public. You may even have seen us make mention of it here, here, here, or even maybe here. Splunkbase was impacted by way of apps both made by Splunk and third-party developers. While my colleagues and I are confident we’ve remediated all of the apps vulnerable to Log4Shell, the endeavor motivated us to take proactive steps to minimize the impact the next Log4Shell, WannaCry, Shellshock, Heartbleed, SolarWinds, or whatever, has on our community. This post is about the enhancements we are making to Splunkbase to ensure a safe, secure, and awesome experience for everyone.

In broad strokes, there are two changes in the works you should know about. The first is that our present policy of allowing third-party developers to create apps for the Splunk Cloud Platform using whatever language stack they choose is going to be refactored. The second is that I and my fellow security engineers are going to be more aggressive in ensuring nothing in Splunkbase has a dependency with a published security defect rated higher than CVE-Medium. On both accounts, the reason for making these changes is to reduce the attack surface presented by the apps hosted in Splunkbase so as to reduce the likelihood the next big cybersecurity incident will have a significant impact on you, our awesome Splunk community.

On the matter of refactoring our policy as to what tech stacks may and may not be used in a Splunk app, we are still finalizing details. In fact, part of the reason for this posting is to give the community an opportunity to weigh in — so don’t be shy about engaging with us, we’d love to hear your thoughts!

Recently I performed an analysis of the contents of Splunkbase using a tool called GuessLang that uses TensorFlow to inspect source code and make an educated guess as to what coding language is in the file. Below are some of the findings. 

GuessLang results when applied to all of Splunkbase, confidence threshold for positive determination set to 0.95/1.0. Note that all GuessLang buckets are represented in this graph, including the ones with zero members. AFAIK, there are no Lisp apps in Splunkbase (sadly). 

Here is the same dataset with the Python bucket removed. As you can see, the next most popular coding stack used in Splunkbase after Python is JavaScript.  

These graphs show two realities of Splunkbase today. The first is that by an enormous margin the apps in Splunkbase are written in Python and JavaScript. The second is that there is a long tail of other stacks like Perl, Java, Ruby, PHP, Go, etc. devs are using to make the magic happen. This long tail represents a security risk in that it complicates to an impossible degree the endeavor of ensuring every usage of every coding stack is adhering to the best security practices for that stack. Moreover, a tiny fraction of developers building apps from Splunkbase are using these other languages, which creates for us a situation wherein our security endeavors in Splunkbase on the whole are being diluted by way of supporting the actions of the few. Therefore, in the future, we are only going to support Cloud Vetting review of apps that employ python and/or JavaScript going forward.

We recognize there may need to be exceptions made to this policy once implemented. There are cases where your Splunk App needs to talk to a third-party API by way of a vendor provided connector. We get it and if that is you then we can talk it out. In general though, we are a data-driven organization and the data strongly indicates that the vast majority of you are already in alignment with this policy change, and that this change will make things faster/cheaper/better for everyone.

When we say "Turn Data into Doing," this is exactly the kind of thing we’re talking about.

As for the second item, being more aggressive about checking apps in Splunkbase for known security defects, you may have noticed both Ryan and Marcus concluded their Log4Shell posts with “Conclusion: Patch Patch Patch” (Ryan’s and Marcus’, respectively). All of us at Splunk Security Operations are in unanimous and emphatic agreement with that sentiment. According to a report by RedScan, NIST security vulnerability trends in 2020: an analysis, there were on average 50 new vulnerabilities published to the National Vulnerability Database and of those ~57% of those were rated ‘critical’ or ‘high’ severity. That was for 2020. CVE details reports the 2021 numbers grew and the 2022 numbers are continuing the upward trend. The single most effective security practice is to aggressively patch your code before someone uses your vulnerable code to create trouble for you and your brand.

Up and to this point we in Splunk security operations have relied on the developer community to keep their apps in Splunkbase patched. Recent analysis of the contents of Splunkbase, resultant from Log4Shell, has shown us the policy isn’t working. Apps tend to get uploaded and forgotten about and forgotten apps tend to accrue security technical debt. To remediate this issue, and be a resource to all of our developers and SplunkBase community, Splunk security operations will:

• Conduct frequent scans of the contents of Splunkbase against the latest vulnerability data from NIVD.
• For any app that is found to have a ‘critical’ or ‘high’ severity vulnerability:
  • The app developer will be notified and given a window of opportunity to patch their app. 
  • Anyone who has downloaded the impacted app will be notified of the issue and, once a patch is available, a link to the patched app.
  • Anyone viewing the app page in Splunkbase will see a security advisory alerting them to the issue.
  • If the developer does not sufficiently patch the app within the window of opportunity, the app will be removed from Splunkbase.

Our goal is to create a transparent process wherein we work with all stakeholders to ensure things are patched in a timely manner, everyone who should be informed is so informed, and apps are not capriciously removed from Splunkbase without cause or due process.

As mentioned before, these are changes that are in process and will be implemented later this year. This first announcement is to give everyone a heads up and an opportunity to give us feedback.

We’re in this together.

We love all of you. 

As always, “be excellent to one another."

^{1 Barrett, Brian. "The Next Wave of Log4J Attacks Will Be Brutal". Wired. ISSN 1059-1028. Retrieved 17 December 2021.}

----------------------------------------------------
Thanks!
David Holiday