My Username Fields Have Passwords in Them! What Do I Do?

Authors and Contributors: As always, security at Splunk is a family business. Credit to authors and collaborators: James Brodsky, Mikael Bjerkeland.

As security practitioners, we like to read blogs, whitepapers, and even Mastodon “toots” that talk about new or novel threats and vulnerabilities. Recently, our fearless and never sleeping Security Strategist Leader James Brodsky called attention to a blog post from the research team over at Mitiga Security that highlighted the risks of password disclosure in authentication logs. None of this was new to us, but what did stand out to us is that this is a perfect opportunity to use Security Orchestration Automation and Response (SOAR) to address this risk in near-real time.

At a very high level, a user might inadvertently enter their password into an authentication prompt, such as the one seen below from Okta:

When this happens, you’re going to have that password recorded, in plaintext, into the Okta System Log that is often fed into Splunk. That’s just the way of the world. This represents something our application security teams refer to as “Insertion of Sensitive Information into Log File” within CWE-532 when done during AppDev. Here, this is not an application flaw or defect, nor is this behavior specific to Okta (it can happen with any authentication provider). However, the searching methods below could be used to identify vulnerable code that is logging and stored within Splunk!

But in this case, we want to find the real account credentials for the user that’s inadvertently put a password into the username field, and do something with the account, preferably through automation. This behavior can be found when hunting through Authentication logs, as one of our awesome Security-savvy Solutions Engineers Mikael Bjerkeland submitted to Enterprise Security Content Update (ESCU). From here, we will need to run a search to identify the suspected user from relevant metadata (e.g., IP, HTTP User Agent) all bounded by time. 

To keep things simple for this example, we used a slightly different search modified from gosplunk:

index=main sourcetype="OktaIM2:log" eventType="user.session.start" 
| eval password=if(match(src_user, "^(?=.*[a-z])(?=.*[A-Z])(?=.*[0-9])(?=.*[\W])(?=.{10,})"), "Yes", "No") 
| stats count by password src_user

But wait! If a user types their password into the username — by definition the resulting log entry doesn’t have the REAL username! How can we find the username that has committed this heinous crime? We need a little more search magic. This time, we’ll look for logins from the same device. If there have been successful logins that contain a real username, we’ll assume that’s the one to further process. Okta happens to fingerprint the devices being used to authenticate, so we can use the value of debugContext.debugData.deviceFingerprint and a second regex to find an email address in the output, and get on with our lives:

index=main sourcetype="OktaIM2:log" eventType="user.session.start" 
| eval password=if(match(src_user, "^(?=.*[a-z])(?=.*[A-Z])(?=.*[0-9])(?=.*[\W])(?=.{10,})"), "Yes", "No") 
| stats list(password) as password,list(src_user) as users by debugContext.debugData.deviceFingerprint 
| search password="Yes" 
| eval user_to_process=mvfilter(match(users,"[\w\d\.\-]+\@[\w\d\.]+"))

Now we’re talkin! The output from a search like the above could create a Notable Event in Splunk Enterprise Security. From there, we could kick this over to a SOAR Playbook.

Our playbook is triggered automatically by the Notable Event being generated in ES. Again, if Okta is our example IAM solution, we could use the SOAR app for Okta to connect to our Okta Workforce tenant. As we’ve determined the suspected user that was affected in the previous step, we need to verify that by polling Okta with the list users action. Next, we’re checking that our user was found in Okta. If they are found, SOAR will follow the left side of the path.

We use a simple code block to generate a password that meets our organization’s password policy. SOAR takes that password and updates the user with it. From there, we send an automated email to the user with the temporary password and then close the SOAR event as well as the Notable Event within ES that started it all off. 

If, for whatever reason, the user account was not found in our IAM system, we follow the right side of the playbook where we update the underlying SOAR event with a new queue and add some notes to explain to the analyst what’s going on. There are many things that we could add or change to this playbook, such as a more secure delivery of the new password, performing a different set of actions if it’s a VIP user, or even telling the IAM system to invalidate all existing sessions and tokens for the user. We could even take action against the event in Splunk by copying it, redacting the password in the src_user field, and placing it in a summary index for further investigation. Then we could delete the original event, so that no unscrupulous users with access to our Splunk instance could harvest those plaintext passwords.

Through the power of automation, you can be proactive in defending your network against common user errors that could turn into a major security incident. To accomplish these kinds of tasks, you can try out our SOAR platform easier than ever these days through the just released Mission Control integration of Splunk ES and SOAR — reach out to your friendly Splunk account team to learn more.