The Holiday Season Is Around the Corner, Are You Ready?

Splunk is committed to using inclusive and unbiased language. This blog post might contain terminology that we no longer use. For more information on our updated terminology and our stance on biased language, please visit our blog post. We appreciate your understanding as we work towards making our community more inclusive for everyone.

With the holiday season quickly approaching — Thanksgiving in the United States, Christmas, New Years, and Lunar New Year in Asia — you need to ensure your alert severity levels are adjusted accordingly for all your services and KPIs. While the current thresholds you set are appropriate for business as usual, these upcoming holidays can result in increased website traffic, message rates, or server usage. To help this, we’re introducing Custom Threshold Windows (CTW), which is the ability to adjust your thresholds for special moments during the year where the regular severity levels don’t apply. Best of all? This was an idea submitted by you through Splunk Ideas.

What Is a Custom Threshold Window and How Do I Use It?

On a “normal” day, the regular severity levels don’t cause false alerts, but during moments of critical increase/decrease in your KPI values (e.g. holidays, monthly patches), the standard thresholding can generate false alerts. Your business cycles are not constant. While adaptive thresholding can help to account for changes in business cycles, it does not account for abnormalities outside of 60-day business cycles.

Let’s walk through how to set up a custom threshold window. First, navigate to Configuration > Custom threshold windows from the ITSI main menus. Next, select Create custom threshold windows to create a new window. A configuration window will appear and allow you to set your customizations. Set your CTW name, description, schedule recurrence, time frame, and duration.

Previously, you had to manually change the thresholds on individual KPIs before a “spike” day happened and then reset the values when the period was over, which is a tedious process. Alternatively, you could use maintenance windows to put the service into maintenance mode, which stopped service/entity-level granularity. With custom threshold windows, you can now set recurring windows — weekly and monthly — as needed.

Now, the fun part. Link at least one KPI to your CTW and increase or decrease the percentage to apply to all selected KPI values. For example, if the range for a Low severity level is set to 10-100, selecting “increase by 10%” updates the threshold to 11-110. 

Once you have saved your CTW, you can edit it any time prior to its initial start. Simply click “edit” on the lister page to add additional KPIs, adjust threshold level percentages, and preview the impacts of the increases and decreases in the preview chart. The preview chart lets you see the current impacts of the thresholds, as well as the impacts across a weekly or monthly span depending on the recurrence you’ve applied.

Congratulations! You have now created your first custom threshold window. With this new feature, you can adjust your levels based on seasonality to fit the needs of your organization. If you require additional support, please refer to the docs. If you have a feature that you’d like to request, feel free to submit it to the Splunk Ideas Portal and encourage your peers to vote on it!