Aw, HEC! Splunking Okta's Event Hooks with the HTTP Event Collector

This weekend I spent some time with Okta’s Identity Engine product, learning about various ways to integrate it with Splunk and other external systems. When I got to Okta’s Event Hooks feature, I exclaimed “Aw, HECk!” (actually I said something a little stronger) and banged my head against my old copy of "Log4J 4 Me and U - A Complete Guide" for a few hours trying to get Event Hooks sending data properly into Splunk’s HTTP Event Collector, or HEC. However, I ultimately prevailed! Here’s a writeup of my experience.

HEC was introduced in 2016 as a high performance, agentless way of getting data into Splunk from services and applications. Any Splunk instance can be used as a HEC endpoint. Tokens are used to secure the service, which optionally uses TLS 1.2. Also, you can use your own certificates to secure the service, which is important for Okta’s Event Hooks to function properly with HEC.

What’s an Event Hook? Well, Okta can send all manner of different events, immediately, via a feature called Event Hooks (very similar to a webhook). These present many different kinds of workflow integration possibilities with any system that can accept output from Event Hooks.

But! There’s a caveat to getting Splunk HEC to work properly here. Okta requires a “one-time verification” step for each Event Hook and in order to do that we need to perform a clever “switcheroo” where we first run a temporary webserver that answers the “verification” POST event from Okta, and then we shut down that webserver and start up HEC.

The basic steps:

1. Set up HEC on your Splunk instance using SSL and choose a TCP port.
2. Install a Node.js environment on your HEC instance. 
3. (Optional) Generate a set of LetsEncrypt certificates, using Node.js and Express to create a temporary webserver that helps you verify you own the domain.
4. Create and start a temporary webserver using Node.js and Express that will respond to the one-time verification challenge using the certificates.
5. Complete the one-time-verification challenge.
6. Stop the temporary webserver, start Splunk HEC, and try sending data.

Let’s get started!

To set up HEC, we first use the GUI in Splunk under Data Inputs, but we’ll need to get into the command line config files before we’re all done. Customers running Splunk Cloud will need to run an on-prem heavy forwarder that communicates up to Splunk Cloud in order to do this because you don’t have access to do the Node.js shenanigans I’m going to lay down below. Or, you could do something clever using name resolution, where you point the URL you ultimately use for HEC temporarily to a different host for the verification step, and then point that same URL to the “real” HEC endpoint.  A good primer for the steps needed to receive webhooks with HEC are within Luke Netto’s blog post here and this Splunk Answers post. The magic configuration step is to set “allowQueryStringAuth=true.”

First, we create a new token under the Data Inputs section of Splunk Settings:

We give it a reasonable sourcetype (I chose okta:eventhook:hec) and tell it what Splunk index to put the data into. You also get a token value, that’s gonna be important later when we set up the Event Hook on the Oktaside, so copy it somewhere.
 

Then we go to Global Settings where we can enable the input. WARNING: enabling this too early will mean that HEC will start listening on the port (I chose 8001, and you also need SSL) and then you won’t be able to start your temporary Node.js webserver later on. So just enable it for testing, and then shut it down.
 

You can test this out and then ensure that HEC is listening on your port in two ways. First, you can do a simple netstat command on the Splunk server running HEC:
 

You can also use curl to send a test event to HEC:
 

curl -k "https://<your server>:<your port>/services/collector/raw?token=<your token>&sourcetype=mydata" -d '1, 2, 3... Hello, world!'

And then search for that event in Splunk:
 

At this point, you know that your HEC is working, so go back into Global Settings and disable HEC for the time being, and make sure it isn’t listening via the netstat command above.. Now, we have to get a Node.js webserver running to complete the Okta one-time-verification step.

First, get your Node.js environment working. My Splunk server is Ubuntu 18.04, so I followed this tutorial to make sure it was installed, and that the npm package manager was working as well. 

Next, we need to generate some actual legitimate certificates from a bona-fide CA so that both our HEC endpoint and our Node.js temporary webserver look reasonable to Okta’s Event Hook service. If you don’t have certificates signed by a proper CA, you’ll get OpenSSL errors when you go to verify. I didn’t have any, so I used LetsEncrypt to make some, And, because I used the “express” Node.js webserver, the tutorial I used to create LetsEncrypt certificates on Ubuntu using Certbot for Express is here. At the end of it, you’ll have all of the requisite files to secure your Express webserver, as well as the HEC endpoint. If you have your own certificates already? This step is optional!

A note for the above, if you’re playing with this in a lab like I was. You must own the domain you’re creating the certificates for during the process. I have “brodsky.cc” as a domain, and the IP address of my Splunk instance is wildcarded, so that something like “splunk1.brodsky.cc” resolves to my Splunk machine running HEC. In order to complete the Certbot steps, I used Node.js to create a simple HTTP webserver (the “first temporary webserver”) on port 80 that served up the static file Certbot was looking for, which I placed in the.well-known directory structure. Here’s that code, which you run with the command line “node server.js” assuming that’s the name of the file. I temporarily opened port 80 to my webserver while the Certbot certificate process was running:
 

// server.js

// // where your node app starts

//

// // Current project dependencies, feel free to add additional libraries or frameworks in `package.json`.

const express = require("express");

const bodyParser = require("body-parser");

const app_request = require("request");

const fs = require("fs");

const app = express();

// serve up the right directory for the Certbot automation to get a file

app.use('/.well-known', express.static('.well-known'));

// https://expressjs.com/en/starter/basic-routing.html

app.get('/test', function (req, res) {

          res.send('my test server')

})

//   // listen for requests :)

//  const listener = app.listen(8001, () => {

//    console.log("Your app is listening on port " + listener.address().port);

//    });

app.listen(80, () => { console.log('listening on 80') });

Since you don’t want an unencrypted webserver running for too long, I’d shut this down as soon as you’re done with it.

OK. Assuming that your Splunk server is accessible from the Internet on the port you’ve chosen to use for HEC (again, I used 8001), you’re ready to take the next step, which is to use the LetsEncrypt (or other) certificate files to start a Node.js temporary webserver and complete the Okta one-time verification. For this, I adapted the code from Heather Wallander‘s Glitch project and blog - specifically her server.js file. Here’s what I came up with. Once it is running, you can hit https://<your webserver IP or name>:<your port>/test to see if it is working.
 

// okta_one_time_verify.js

// // where your node app starts

//

// // Current project dependencies, feel free to add additional libraries or frameworks in `package.json`.

const express = require("express");

const bodyParser = require("body-parser");

const app_request = require("request");

const fs = require("fs");

const app = express();

// change the three lines below to point to your letsencrypt files

const key = fs.readFileSync('/etc/letsencrypt/live/brodsky.cc/privkey.pem');

const cert = fs.readFileSync('/etc/letsencrypt/live/brodsky.cc/cert.pem');

const ca = fs.readFileSync('/etc/letsencrypt/live/brodsky.cc/chain.pem');

const https = require('https');

const server = https.createServer({key: key, cert: cert, ca: ca}, app);

// from Heather Wallander project to respond to Okta verification challenge

app.get("/services/collector/raw", (request, response) => {

          var returnValue = {

                      "verification": request.headers['x-okta-verification-challenge'],

                    };

          response.json(returnValue);

});

// https://expressjs.com/en/starter/basic-routing.html

app.get('/test', function (req, res) {

          res.send('my webserver is running')

})

server.listen(8001, () => { console.log('listening on 8001') });

Now, you can run the one-time-verification step. Start your temporary webserver like so (I called this file okta_one_time_verify.js):
 

Then, fill out the Event Hook dialog in Okta like so, adjusting the URL to point to your server running HEC, and adjusting the token to include what you saved way up near the beginning of this blog. I’m testing using the “User created” event, which means every time a user is created in Okta, I’ll immediately get a post of that event to Splunk HEC.
 

An example full URL string is:
 

https://<your domain>:8001/services/collector/raw?token=<your token string>

Once you save the above, you’ll be presented with this dialog:
 

So click Verify, and if all goes well, you’ll see this:
 

Almost done! Shut down your temporary Node.js webserver, and start Splunk HEC again. But before you do that, you’ll want to secure HEC with the same certificates we generated above. Follow this Splunk Answers post which explains how to do that. The last step is to verify that you can send actual Event Hook content to HEC. You do this using the Preview function:
 

Click the “deliver request” button at the bottom of the page, and you should see “Request Delivery Successful”:
 

To confirm data in Splunk, run a search like “index=main sourcetype=”okta:eventhook:hec”:
 

If you see output like the above, you have successfully delivered Okta Event Hook data into Splunk’s HEC, securely!

I hope this little tutorial gets you further down the path of consuming Okta Event Hook data in Splunk. Please direct any questions about it to me at @james_brodsky on Twitter or hit me up on LinkedIn. 

Happy Event Hook Splunking!