How to Create a Modular Alert

What’s a Modular Alert (and why should I care)?

Modular Alerts is a feature in included in Splunk 6.3 and later that allows it to actively respond to events and send alerts, gather more data, or perform actions. Splunk includes an API that makes it easy for people to write their own apps with modular alerts that can be shared on apps.splunk.com.  See the official docs for more detailed information.

Modular Alerts can used for things such as:

• Notifications: send out a message letting people know something happened (e.g. Twilio SMS Alerting, Slack Notification Alert, HipChat Room Notification Alert(
• Automation: perform an action whenever a particular event is detected by Splunk (e.g. Insteon Home Automation Control, IFTTT Alert Action, Octoblu Alert Action, Philips Hue Alert Action)
• Information gathering: gather more information whenever something happens

In this post, I’ll walk you though how to write a Modular Alert. The entire example app source code is posted on Github here and the installable example app is here.

Alright, I want to make one

Ok, glad you agree partner. Let’s do this. We are going to make a modular alert that just logs something inside Splunk. This example won’t do much but you should be able to see how to expand this into a new modular alert of your own from this example.

What are the parts?

Before we get started, lets review the components of a Modular Alert. Below is the list of files that an app should include for a functioning Modular Alert:

•  README/
  • alert_actions.conf.spec (declares the alert action and defines the supported parameters)
• bin/
  • <modular_alert_name>.py (the Python class containing the code executed by the modular alert)
• appserver/
  • static/
    • appIcon.png (the app icon, can be used as the icon for the modular alert)
• default/
  • setup.xml (optional, a setup page for configuring global default values for the modular alert)
  • alert_actions.conf (defines details about the modular alert and app default values for the parameters)
  • data/ui/alerts/
    • <modular_alert_name>.html (an HTML template defining the UI of the modular alert configuration page)

Step 1: make a basic app

To start, lets make a basic app. This will consist of an app.conf file that describes your app. The app.conf file will go under the default directory. Completing the app.conf is fairly intuitive but make sure to check out the documentation if you need help. See below for the content of the app.conf for this sample app:

[launcher]
version = 1.0
description = An example of a Splunk modular alert
author = LukeMurphey

[package]
id = splunk_modular_alert_example

[ui]
is_visible = false
label = Splunk Modular Alert Example

The app is set to be invisible (is_visible=false) because it doesn’t include views and thus it doesn’t need to be included in the list of apps on Splunk home page. I also included some meta-data that gives everyone read access and gives administrators write access. The default.meta will share your alert action globally. Without it, users will be restricted to using the alert action within this app (e.g. you wouldn't be able to use the alert action in the Search app). 

Once this is all done, our app will include the following files:

• modular_alert_example/
  • metadata/
    • default.meta
  • default/
    • app.conf

See the change here in more detail.

Deploy this onto a Splunk box and restart it. Once you do, you should see the app listed in the list of installed apps (look for “Splunk Modular Alert Example”):

Step 2: make the alert action conf files and the icon

Now, lets start the process of making the Modular Alert. To do so, we will need to make two files: alert_actions.conf and alert_actions.conf.spec.

Make alert_actions.conf.spec

The alert actions.conf.spec file will describe our alert action to Splunk in order to define the fields that the alert action expects. The file needs to be placed under the README directory. This results in the following files in our app:

• modular_alert_example/
  • metadata/
    • default.meta
  • default/
    • app.conf
  • README/
    • alert_actions.conf.spec

The file itself will include the list of fields under a stanza which indicates the name of the alert action. This modular alert action is going to just going to log a message, thus, I’ll call it “make_a_log_message”. It takes two parameters, a message to log and a value indicating the importance. This results in the following:

[make_a_log_message]
param.message = <string>
param.importance = <integer>

See the change here in more detail.

Make alert_actions.conf

Next, make the alert_actions.conf file. See the spec for alert_actions.conf for details. The file will go under the default directory. This results in the following:

• modular_alert_example/
  • metadata/
    • default.meta
  • default/
    • app.conf
    • alert_actions.conf
  • README/
    • alert_actions.conf.spec

The stanza for the alert action needs to match the stanza name in the alert_actions.conf.spec file you previously created. The label field is used by Splunk when it lists the available modular alerts and the description provides more details on what it does. The icon_path parameter tells Splunk where the icon is that should be used to represent the alert action. In this case, we are going to use the app icon (appIcon.png). The param fields are used to provide a default value for the alert action. In this case, I am providing a default importance of 0 (zero). The file looks like this:

[make_a_log_message]
is_custom = 1
label = Make a log message
description = Makes a log message in the _internal index (an example of a modular alert)
icon_path = appIcon.png
payload_format = json

# Default value for importance
param.importance = 0

See the change here in more detail.

Make an icon

Now, lets make an icon so that the list of alert actions look a little nicer. To do this, I’ll include a 36×36 PNG file in appserver/static. This results in the following:

• modular_alert_example/
  • metadata/
    • default.meta
  • default/
    • app.conf
    • alert_actions.conf
  • README/
    • alert_actions.conf.spec
  • appserver/
    • static/
      • appIcon.png

In this case, I’m going to use this icon both for the app icon as well as the icon for the alert action. Make a different icon (e.g. “appserver/static/alertAction.png”) if you want to use different files.

See the change here in more detail.

Deploy this onto a Splunk box and restart it. Then, navigate to “Alert actions” in the “settings” menu at the top right of the Splunk web UI. You should see your alert action listed (look for “Make a log message”):

Step 3: make the modular alert configuration view

Splunk gives you the ability to define an HTML stub that provides a UI for editing your alert action entries. This HTML stub will be rendered when someone configures your alert action for a saved search. To make this page, create a file under default/data/ui/alerts/ with the file name “make_a_log_message.html”. The file name needs to match the stanza name of the alert action.

This results in the following:

• modular_alert_example/
  • metadata/
    • default.meta
  • default/
    • app.conf
    • alert_actions.conf
    • data/
      • ui/
        • alerts/
          • make_a_log_message.html
  • README/
    • alert_actions.conf.spec
  • appserver/
    • static/
      • appIcon.png

See the change here in more detail.

Deploy this onto a Splunk box and restart it. Then, view the dialog by doing the following:

1. Open the search view and run some search (e.g. “* | head 1”)
2. Click “Save As” > “Alert”
3. Click “Add Actions” and select “Make a log message”

You should see your alert action configuration page:

Step 4: make the modular alert Python class

Thus far, we have registered an alert action but we haven’t enabled it do anything. In this step, we will fill in the code to make it do something.

Get the modular alert base class

I wrote a class that simplifies the creation of a Modular Alert. You can download it from Github. The license is intentionally permissive so that you can use it in your own apps (even paid ones). Place this file under the directory bin/modular_alert_example_app. Additionally, make an empty file named “__init__.py” in the directory so that Python will treat this as a module.

This results in the following:

• modular_alert_example/
  • metadata/
    • default.meta
  • default/
    • app.conf
    • alert_actions.conf
    • data/
      • ui/
        • alerts/
          • make_a_log_message.html
  • README/
    • alert_actions.conf.spec
  • appserver/
    • static/
      • appIcon.png
  • bin/
    • modular_alert_example_app/
      • __init__.py
      • modular_alert.py

See the change here in more detail.

Create your modular alert class

Now, lets make the class that will do the work. To do this, make a Python file named make_a_log_message.py under the bin directory.

This results in the following:

• modular_alert_example/
  • metadata/
    • default.meta
  • default/
    • app.conf
    • alert_actions.conf
    • data/
      • ui/
        • alerts/
          • make_a_log_message.html
  • README/
    • alert_actions.conf.spec
  • appserver/
    • static/
      • appIcon.png
  • bin/
    • modular_alert_example_app/
      • __init__.py
      • modular_alert.py
    • make_a_log_message.py

Start populating your make_a_log_message.py with some imports:

import logging
import sys
from modular_alert_example_app.modular_alert import ModularAlert, Field, IntegerField, FieldValidationException

See the change here.

This will import the base class and some classes we will need to make our alert work. I am importing the Field and IntegerField classes because the modular input will need to validate the string field “message” and the integer field “importance”.

Next, add the class for your modular input that sub-classes. Here I have made a class which inherits from ModularAlert and implements a constructor. The constructor includes a list of the parameters that the alert action expects (based on the contents of the alert_actions.conf.spec file). This allows the Python code to make sure that the incoming values are valid and it will convert them to the appropriate Python object. For example, the importance field will be automatically converted to an integer since it using the IntegerField class. Make sure that your constructor calls the super class constructor too. Don’t worry about the contents of the run function yet, we will fill that out next.

import logging
import sys
from modular_alert_example_app.modular_alert import ModularAlert, Field, IntegerField, FieldValidationException

class MakeLogMessageAlert(ModularAlert):
    """
    This alert just makes a log message (its an example).
    """
    
    def __init__(self, **kwargs):
        params = [
                    IntegerField("importance"),
                    Field("message")
        ]
        
        super(MakeLogMessageAlert, self).__init__(params, logger_name="make_a_log_message_alert", log_level=logging.INFO )
    
    def run(self, cleaned_params, payload):
        pass

See the change here.

Next, populate the run function with the code to make your modular alert do something. The run function will be called with the your parameters in the “cleaned_params” argument. These arguments will already be converted to Python objects (e.g. the importance field will be an integer). I am using the get() function with a default value as the second argument so that I can define a default value in case that parameter wasn’t provided. You can use the logger instance in self.logger in order to post information about what your modular input is doing. This is important in order to aid debugging in case something goes wrong. This results in:

import logging
import sys
from modular_alert_example_app.modular_alert import ModularAlert, Field, IntegerField, FieldValidationException

class MakeLogMessageAlert(ModularAlert):
    """
    This alert just makes a log message (its an example).
    """
    
    def __init__(self, **kwargs):
        params = [
                    IntegerField("importance"),
                    Field("message")
        ]
        
        super(MakeLogMessageAlert, self).__init__(params, logger_name="make_a_log_message_alert", log_level=logging.INFO )

    def make_the_log_message(self, message, importance):
        """
        This is the function that does what this modular alert is supposed to do.
        """
        self.logger.info("message=%s, importance=%i", message, importance)

    def run(self, cleaned_params, payload):
        
        # Get the information we need to execute the alert action
        importance = cleaned_params.get('importance', 0)
        message = cleaned_params.get('message', "(blank)")
        
        self.logger.info("Ok, here we go...")
        self.make_the_log_message(message, importance)
        self.logger.info("Successfully executed the modular alert. You are a total pro.")

See the change here.

Finally, you will need to add some boilerplate code that will get your modular input to execute in Splunk. This code makes an instance of your class and gets it to communicate with Splunk over standard input:

import logging
import sys
from modular_alert_example_app.modular_alert import ModularAlert, Field, IntegerField, FieldValidationException

class MakeLogMessageAlert(ModularAlert):
    """
    This alert just makes a log message (its an example).
    """
    
    def __init__(self, **kwargs):
        params = [
                    IntegerField("importance"),
                    Field("message")
        ]
        
        super(MakeLogMessageAlert, self).__init__(params, logger_name="make_a_log_message_alert", log_level=logging.INFO )

    def make_the_log_message(self, message, importance):
        """
        This is the function that does what this modular alert is supposed to do.
        """
        self.logger.info("message=%s, importance=%i", message, importance)

    def run(self, cleaned_params, payload):
        
        # Get the information we need to execute the alert action
        importance = cleaned_params.get('importance', 0)
        message = cleaned_params.get('message', "(blank)")
        
        self.logger.info("Ok, here we go...")
        self.make_the_log_message(message, importance)
        self.logger.info("Successfully executed the modular alert. You are a total pro.")
        
        
"""
If the script is being called directly from the command-line, then this is likely being executed by Splunk.
"""
if __name__ == '__main__':
    
    # Make sure this is a call to execute
    if len(sys.argv) > 1 and sys.argv[1] == "--execute":
        
        try:
            modular_alert = MakeLogMessageAlert()
            modular_alert.execute()
            sys.exit(0)
        except Exception as e:
            print >> sys.stderr, "Unhandled exception was caught, this may be due to a defect in the script:" + str(e) # This logs general exceptions that would have been unhandled otherwise (such as coding errors)
            raise
        
    else:
        print >> sys.stderr, "Unsupported execution mode (expected --execute flag)"
        sys.exit(1)

See the change here.

Deploy this onto a Splunk box and restart it. Then, make a search and add your alert action so that you can test it. You should see the output once your search executes. You can then view the results of the alert actions execution by running the following search:

index=_internal sourcetype=splunkd component=sendmodalert action="make_a_log_message"

If you do not see output, run the following search and look for errors:

index=_internal sendmodalert sourcetype=splunkd

Step 5: make a setup page to set global defaults (optional)

Optionally, you can create a setup.xml page that sets default values for the Modular Alert. This useful for configuration items such as authentication which you may not want to have repeated for every instance of the alert action. This can also be used to modify the default values presented when configuring a modular alert.

To make a setup page, create a file named “setup.xml” in default/setup.xml. This results in the following file-structure:

• modular_alert_example/
  • metadata/
    • default.meta
  • default/
    • app.conf
    • alert_actions.conf
    • setup.xml
    • data/
      • ui/
        • alerts/
          • make_a_log_message.html
  • README/
    • alert_actions.conf.spec
  • appserver/
    • static/
      • appIcon.png
  • bin/
    • modular_alert_example_app/
      • __init__.py
      • modular_alert.py
    • make_a_log_message.py

New populate the setup.xml file with a setup page. See the spec file for details. In this case, I just want a setup page that edits the “message” parameter of the “make_a_log_message” alert action. The setup page ends up looking like this:

<setup>
  
  <block title="Global Settings for the Alert Action" endpoint="admin/alert_actions" entity="make_a_log_message">
	  
      <text>An example of a setup page for the make-a-log-message example modular alert action
      </text>

      <input field="param.message">
        <label>Enter a default message</label>
        <type>text</type>
      </input>
      
  </block>

</setup>

See the change here.

Deploy this onto a Splunk box and restart it. The Alert Action list in the Splunk’s Manager will now show a link to your setup page (see the link titled “Setup Splunk Modular Alert Example”):

If you click the link to setup up the app then you will be taken to the setup page. I modified the default message to be “This is the message set in setup.xml”:

If you save the change, then Splunk will set the default value to whatever you defined. This deploys a local version of alert_actions.conf (in /modular_alert_example/local/alert_actions.conf) which includes the following in the example above:

[make_a_log_message]
disabled = 0
param.message = This is the message set in setup.xml

Once the default is change, any new alert actions will have this value in the message field by default. If I create a new instance of the alert action, I see the message field is already populated with the value I set above:

Conclusion

That’s basically it. Consider submitting your app to apps.splunk.com if you make an app so that others can use it too. Let us know if you run into problems and post a question on Splunk answers if you get stuck.

----------------------------------------------------
Thanks!
Luke Murphey