Updated on Oct 8, 2018 for new password parameters

Original post from January 17, 2018:

This is the first in a series of posts covering Splunk-related activities you can do from the comfort of your own...workstation.

This particular topic was presented at .conf2017 as a Hands on Lab by Burch entitled Sandboxing with Splunk (with Docker).

Prerequisites

• Make sure you are comfortable with Splunk Enterprise: Before you jump in, you should be comfortable installing Splunk Enterprise, starting it from the command line, and the usage of network Ports by Splunk. We're gonna be referring to some stuff in those domains that will absolutely confuse you if it's the first you're hearing of them.
• You do NOT need to be a Docker expert: One thing you don't need to be is a Docker expert. In fact, I'm NOT a Docker expert at all! I'm just so happy with this idea that I couldn't help but want to share it, my poor Docker skillz notwithstanding.

Further Reading

You can use this blog to get started with Sandboxing and Docker without any prior experience or knowledge. However, if you want to learn more you can check out .conf2016's Your Splunk Sandbox and .conf2017's Sandboxing with Splunk (with Docker). Both of these presentations are great background material.

Why Bother with a Sandbox?

Usually, someone who needs a sandbox has been saying things like:

"I'd like to get to the bottom of odd phenomenons and unexpected behavior, but I'm scared to break my environment."

"The more work I put into my environment, the more hesitant I am to try new things."

"I'd like to keep learning and experimenting."

"I need a proper testing environment to try out new ideas."

"I'd like an environment where I can just see what happens, without worrying about crashing."

Sandbox Tips

• You don't HAVE to use Docker. 
• Any Sandbox that you're comfortable destroying will suffice. In .conf2016's Your Splunk Sandbox, I share a few options and considerations.
• Make sure everyone has their own individual sandobx. A 'team sandbox' is inheritly flawed. If your team is sharing the sandbox, they will all be afraid to make changes that could impact each other. This will stifle their learning.

Setting Up Your Workstation

I suggest using your local workstation (laptop/desktop) machine. While you certainly can use a remote host, you should recognize that you may be introducing too much complexity. This burden of use could discourage you from using the sandbox altogether. If you are using Docker, it is much preferred to host the Docker environment locally. I would argue that the costs (e.g. disk space, admin exception) are well worth the benefits.

Download & Install Docker

Navigate over to Docker's official website and follow their instructions for downloading and installing docker on your target machine. At the time of this writing, there's a "Get Docker" menu on top of the page that will get your started and pass you along to a few different pages before you get the download going.

Remember, Docker is a different company and different product than Splunk. So if you run into problems with this part (navigating, downloading, and installing) you'll want to peek at the Docker documentation and/or work with Docker, not Splunk. OK, back to the fun!

Get the Splunk Enterprise Docker Image

The Splunk Enterprise Docker image is hosted on the Docker Store. Unlike the online stores we're used to, there's nothing to download on that particular page. Instead, you'll see instructions including the docker pull command for having your Docker install fetch and download the Splunk Enterprise Docker image. Follow those instructions and soon enough you'll see the component pieces of the image being downloaded. It'll look something like this:

$ docker pull splunk/splunk
Using default tag: latest
latest: Pulling from splunk/splunk
ad74af05f5a2: Pull complete 
6ed26c881126: Pull complete 
0efc5eeb5075: Pull complete 
123d19a3ee15: Pull complete 
6fe48f1452ee: Pull complete 
fc6bbc9992f6: Pull complete 
8ebdf9134129: Pull complete 
Digest: sha256:1be3208a6c1d96ca5ad320fc21cbfcf06428e3ea12f10773e2efc7d2dbb4b522
Status: Downloaded newer image for splunk/splunk:latest

You can even validate what images you have with docker images.

splunk/splunk 

latest              

09d63966df06        

5 weeks ago

758M

Managing Splunk Docker Instances

So...now you have Docker installed and you've got the Splunk Enterprise image. Next up, creating instances of Splunk Enterprise!

Important Note
In Docker, instances are referred to as containers. Throughout the rest of this blog post, the terms instance and container will be used interchangeably (they're the same thing).

Creating a Splunk Docker Instance

Here's the simplest command to start up a new Docker container (instance) of Splunk Enterprise:

docker run -P -d -e 'SPLUNK_START_ARGS=--accept-license' -e 'SPLUNK_PASSWORD=changeme' splunk/splunk

But let's break down the parameters in this use of the docker command by showing the parameters description from docker help run along with its relevance for the Splunk image.

run

-P

-d

-e

splunk/splunk

Connect to the Splunk Docker Instance

Great! So...where is that Splunk instance we just created? Well, this is where the docker container command comes in handy. For example, if I run docker container list -a I'll see the following columns in my output. Here's the relevance of each of these headers:

CONTAINER ID

IMAGE

COMMAND

CREATED

STATUS

PORTS

NAME

Ports

OK, I promised I'd elaborate on that one, since the notation might be a bit new to you. But fear not! It's really quite simple.

You'll probably see something like this, but with different numbers to the left of each '->':

0.0.0.0:32779->1514/tcp, 0.0.0.0:32784->8000/tcp, 0.0.0.0:32777->8088/tcp, 0.0.0.0:32776->8089/tcp, 0.0.0.0:32775->8191/tcp, 0.0.0.0:32774->9997/tcp

If you've gone cross-eyed, take a step back. You should see that we've just got ourselves a collection of port pairings...and you might even notice some of the right-hand side ports look very Splunk-y. What happened here is Docker has made those ports available for you to access from your machine, but randomly assigned to different ports.

This reassignment might seem annoying if you're used to installing Splunk on your local machine but trust me, it is a huge benefit! Imagine having dozens or hundreds of Splunk containers all running at the same time. You don't need to mess with port conflicts! The instances can all run simultaneously without you having to administer different ports! Boooya!

Now let's look at each individual pairing. We've got:

0.0.0.0:<port>-><splunk default port>/<protocal>

It's saying that if you go to the left side of the arrow (->) Docker's networking mapping magic will forward those requests to the port listed on the right side of the arrow as the specified protocol. Let's look at the following example:

0.0.0.0:32784->8000/tcp

If we put 0.0.0.0:32784 in a browser (or localhost:32784), Docker will send those requests to the container's service listening on port 8000 as tcp. Since SplunkWeb's default port is 8000, you'll see something like this:

Editing *.conf Files

While you can certainly use SplunkWeb to make changes to the environment, you'll probably get to a point where you want to manually edit .conf files or save your work from the container. This is where things get a bit annoying. That's because the Docker container doesn't have much installed out-of-the-box, not even vi/vim! So, while you can explore the container's terminal using docker exec -it <container name|id> bash, you might not want to given other options.

I probably sound a bit crazy right now, but the reality is that in order to be successful we need to keep our work entirely separate from our sandboxes. This will allow us to destroy our sandboxes and rebuild without a worry in the world. With that in mind, I prefer to mount a folder from my desktop into the container as a Splunk app. By mounting that folder as a volume, I can manage the contents (add/remove/edit files) with my preferred navigator (Mac Finder, Windows Explorer, terminal, etc...) and my preferred editor (vi, SublimeText, Notepadd++ etc...). Since the folder is mounted as a volume, the changes I make locally are reflected within the container.

The syntax for this is the -v option when I first instantiate the container. docker help run informs us that this parameter is used to Bind mount a volume. An example of the syntax is if we insert within the run command:

…license" -v ~/Desktop/local_app:/opt/splunk/etc/apps/container_app splunk/s…

In this example, local_app is the folder on my system and container_app is how it appears on the container's filesystem. Notice that it's slipped into the docker help run between other parameters but before we call out the image (splunk/splunk).

Don't forget that like with other Splunk config changes, you'll want to restart for the changes to take affect. While you can use the WebUI restart controls, you can also apply a restart by restarting the docker container with docker stop <container_name|id> or docker start <container_name|id>.

Docker Command Review

Phew, that's a lot. Let's pause here and review a cheat sheet I put together of what we've covered thus far. _{(Fair warning: The URL in the subtitle may no longer be accessible)}

Challenge

Ok, let's apply everything we learned to appreciate the benefits of using Docker for Splunk sandboxes. Try the following tasks. If you get stuck, the solution column should help you out. 

docker run -P -d -e 'SPLUNK_START_ARGS=--accept-license' -e 'SPLUNK_PASSWORD=changeme' splunk/splunk

docker rm -vf <container_name|id>

docker run --hostname fun -P

docker run --name fun -P

docker run -P -d -e 'SPLUNK_START_ARGS=--accept-license' -e 'SPLUNK_PASSWORD=changeme' -e "SPLUNK_USER=root" splunk/splunk

docker run -P -d -v ~/Desktop/my_app:/opt/splunk/etc/apps/my_app -e 'SPLUNK_START_ARGS=--accept-license' -e 'SPLUNK_PASSWORD=changeme' splunk/splunk

Next Steps

Congratulations! You've gotten your basics on how to use Docker as your Splunk Enterprise sandbox. We're all done here and I hope you enjoyed this. If you want to learn more, here's some resources worth checking out. Enjoy and happy Splunking!

• Container Resources: You can adjust the computing resources available to Docker. This allows your containers to use CPU and Memory. The Mac Download Page shows some of those tweaks available in the Docker preferences.
• Do More with Splunk Container: Don't forget that there's a TON more settings you can tweak listed on the Splunk Enterprise Image in the Docker Store.
• Data Volume License Increase: Your Splunk instances will be using the free license. Remember that if you're violating the license, it's probably easier to just destroy and create a new Docker container than it is to implement a Development License. Remember that if you are afraid to destroy your container because you put work into then you likely could be Sandboxing better: separate the work from the sandbox so you can rebuilt without pain!
• Get Sample Data: Some options with increasing complexity...
  1. Use the _internal index on the instance and the data already in there
  2. Fetch fake data from our Search Tutorial
  3. Export Data from another Splunk Environment
  4. Make your Docker instance a Search Head of an Indexer with data already on it
  5. Use the Splunk Event Generator
• Upgrade Splunk (Image): When it's time to upgrade your Splunk image, simply run docker pull splunk/splunk:latest. Don't bother upgrading the containers themselves. If you are Sandboxing correctly, it should be no problem to destroy the old containers and create new ones on the newer Splunk Enterprise image release.
• Universal Forwarder: Did you notice that the Splunk Universal Forwarder is also available as an image on the Docker Store?
• Splunk n' Box: Take Docker to the next level with Mo Hassan's Splunk n' Box where you can instantly create complicated distributed Splunk deployments with Docker on your local workstation!