Introducing the Splunk Infrastructure Monitoring Terraform Provider

Splunk Infrastructure Monitoring is proud to announce that, with the help of our friends at HashiCorp, our Terraform provider is now an official Terraform provider. This means that you and your organization can use our provider merely by referencing it in your Terraform files and running terraform init. You no longer need to manage compiling and distributing it on your own!

How We Got Here

In the heady days of 2016 the fine folks at Yelp created their own Terraform provider for Splunk Infrastructure Monitoring called SignalForm and made it available to the world. As Terraform gained popularity, this work was beneficial to everyone. Jump forward to 2018 and Stripe forked Yelp’s work to take it further. This work made it clear that Splunk Infrastructure Monitoring's users would benefit from official support by Splunk Infrastructure Monitoring. With help from HashiCorp — the creators of Terraform — we worked through a review process and are proud to share a revamped, thoroughly tested, and officially supported project to share with you. We’d also like to thank both Yelp and Stripe for their effort and support.

Monitoring As Code

HashiCorp Terraform is a provisioning tool that enables organizations to adopt an infrastructure as code workflow. Using configuration files stored in version control your organization can manage in-house assets or infrastructure in one — or even many — clouds.

Since monitoring — dashboards, alerts and more — is a part of your infrastructure, it is helpful to manage them in a similar way. This gives us monitoring as code and provides similar benefits like automation, visibility, and collaboration. By leveraging Git repositories of configuration, large swaths of monitoring assets can be created quickly, managed using standard tools like Terraform, and worked on by anyone in the organization who can send a pull request.

Terraform is pretty popular, but it isn’t the only way to make use of Splunk Infrastructure Monitoring's extensive API for monitoring as code. Our friends at Nike made the signal_analog tool for similar reasons.

When Should You Manage Splunk Infrastructure Monitoring with Terraform?

You might be wondering how to choose between using Terraform for dashboards and detectors versus managing them in the UI. Great question! Splunk Infrastructure Monitoring has invested a lot of time and effort into our UI and we think it’s generally the best way to build and maintain these assets. There are even tools like mirrored dashboards to help reuse! But in some situations Terraform makes a lot of sense:

• Having Git history for things is great!
• Some DevOps heavy organizations like to colocate alerts with code, and keep them in the same Git repo.
• Keeping detectors in Git with Terraform can facilitate greater collaboration, allowing easy pull-request changes that are reviewed by the whole team.
• Why not both? Authoring detectors in the UI can speed things up, then you can copy and paste the resulting SignalFlow into a Terraform file or terraform import and terraform show state to “export” the detector.
   

Whatever you choose, our Terraform provider is here to help.

Quick Tutorial

Your organization may already be using Terraform for managing AWS or GCP things. If not, HashiCorp has excellent tutorial resources you can look into to get the basics like installation. From the basics, using Splunk Infrastructure Monitoring's provider is really easy! Let’s make a quick detector. We’ll start with an empty directory and make a new file called `main.tf`:

```
provider "signalfx" {
  auth_token     = "XXXADDTOKENHERE"
  # If you use SSO or a custom domain, add this!
  # custom_app_url = "https://yourorg.signalfx.com"
}
# Every resource needs a unique name
resource "signalfx_detector" "application_delays" {
    name = "Customer latency is high"
    description = "SLI metric for customer experienced latency is higher than expectations"
    program_text = <<-EOF
signal = data('app.delay_seconds').max()
detect(when(signal > 2, '1m')).publish('Latency High')
    EOF
  rule {
        detect_label = "Latency High"
        description = "Latency is high for the last minute"
        severity = "Critical"
        #notifications = ["Teamabc123"] # Team ID goes here, send em a message!
  }
}
```

From that directory you can now run `terraform init` to set up Terraform’s infrastructure and install the Splunk Infrastructure Monitoring plugin. Next, run `terraform plan` and be greeted with a “plan” showing what Terraform plans do for you. Remember, this is just a plan, no action is taken yet!

```
An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  + create
Terraform will perform the following actions:
  + signalfx_detector.application_delays
      id:                                    <computed>
      description:                           "SLI metric for customer experienced latency is higher than expectations"
~ rest of this is elided ~
```

Great, that’s exactly what we wanted! You can now run `terraform apply`, confirm the plan, and say “yes” at the prompt.

```
Apply complete! Resources: 1 added, 0 changed, 0 destroyed.
```

You can now check out this detector in Splunk Infrastructure Monitoring and see your handiwork. Note: detectors made via the API look a bit different than those made via the UI for now. More on that another time!

Let’s imagine that we now need to make a change. We’ve decided to tighten up our SLA and alert when the delay is higher than 1 second. No problem just edit the main.tf file to `> 1` and run `terraform plan`:

```
Terraform will perform the following actions:
  ~ signalfx_detector.application_delays
      program_text: "signal = data('app.delay_seconds').max()\ndetect(when(signal > 2, '1m')).publish('Latency High')\n" => "signal = data('app.delay_seconds').max()\ndetect(when(signal > 1, '1m')).publish('Latency High')\n"
Plan: 0 to add, 1 to change, 0 to destroy.
```

Again, that’s exactly what we want. Now we can run `terraform apply` and our detector will dutifully change.

```
Apply complete! Resources: 0 added, 1 changed, 0 destroyed.
```

We’ve only touched the surface of what you can manage with the provider, so please check out our documentation for more.

Get Started With Splunk Infrastructure Monitoring and Terraform

If you’re new to HashiCorp Terraform, we hope this whets your appetite to learn more and to try out our provider. If you’re already Terrafirmly using Terraform then we hope you enjoy the improvements we’ve been making and encourage you to shoot us suggestions, fixes or ideas. You can file bugs or suggestions on GitHub at terraform-providers/terraform-provider-signalfx.

Monitoring as code is great, but we’ll also take this opportunity to plug the Splunk Infrastructure Monitoring API as a whole and remind you that Terraform is just one way to leverage that power. You can even run a query over a websocket and stream the results back for cryin’ out loud!

If you’re not already using Splunk Infrastructure Monitoring, get started today with our 14-day free trial. 

----------------------------------------------------
Thanks!
Cory Watson