Change Management for SOA Configuration

In a previous blog entry, I had mentioned that Splunk can participate as a Services Oriented Architecture (SOA) consumer and provided an example on using web services as a scripted input. In today’s entry, I’ll discuss a more administrative task, which is quite native to Splunk, change management. As you may well know, Splunk can audit the file system and provide events on any change to a directory such as additions, updates, and deletes. Splunk can even monitor the files’ contents so that you can do a Splunk diff command to know exactly what has changed.

Now, what does this have to with SOA? In a typical SOA set up, there will exist a number of configuration files. At a minimum, you may have Web Services Definition Language (WSDL) files, XML files, and XML Schema (XSD) files stored on the file system of a production machine. It is important that any change that occurs to these files be authorized and monitored to provide control over deployment.

Let me introduce a simple use case. First, use Splunk’s fschange as a data input in your inputs.conf to monitor a directory. For example,

[fschange:/Applications/splunk/wsdl]
sourcetype = wsdl_monitor
index = testing
disabled = false
_blacklist = [~]$
_whitelist = \.wsdl$
recurse = true
sendEventMaxSize = -1
pollPeriod = 600
fullEvent = true

This stanza means I am monitoring my WSDL directory recursively for changes to WSDL files (excluding backup files ending with ~) every 600 seconds and I would like the full event (the entire file) to be stored within Splunk. Notice, I have an index = testing criteria, because the default is that index = _audit, which may not be where you want to place file system change events. You can also use index = main if you want your main index to have these events. Now, after I restart Splunk, I can start making changes to my configuration files and Splunk will tell us what has changed.

My search index=testing sourcetype=”wsdl_monitor” yields results such as:

Mon Mar 16 18:22:06 2009 action=update, path="/Applications/splunk/wsdl/iptocountry.wsdl", isdir=0, size=9614, gid=20, uid=503, modtime="Mon Mar 16 18:21:58 2009", mode="rw-r--r--", hash=, chgs="modtime "

Mon Mar 16 18:19:03 2009 action=add, path="/Applications/splunk/wsdl/iptocountry.wsdl", isdir=0, size=9599, gid=20, uid=503, modtime="Mon Mar 16 18:18:02 2009", mode="rw-r--r--", hash=

Mon Mar 16 18:09:49 2009 action=delete, path="/Applications/splunk/wsdl/WeatherService.wsdl"

I can now tell who changed the file or directory and when it was done. Notice how action=add, action=update, and action=delete can be used as the basis to form event types which can later be used for Splunk Alerts. This means that changes to the SOA configuration can actively be monitored. Moreover, if a change was supposed to happen, and a search yielded no results for your action=update path=/Applications/splunk/wsdl/iptocountry.wsdl within the last 24 yours, the absence of a change may be worthy of an alert as someone forgot to update the WSDL.

An interesting next step is to see what changes were made to a configuration file to see if the changes were innocuous. In my example, the search index=testing source=”/Applications/splunk/wsdl/iptocountry.wsdl”|diff yields the following:

diff x y compares x to y
- indicates a line present in x but missing in y
+ indicates a line present in y but missing in x
! indicates a line that exists in both x and y, but contains different information

   </wsdl:message>
   <wsdl:portType name="iptocountrySoap">
     <wsdl:operation name="FindCountryAsString">
-      <wsdl:documentation xmlns:wsdl="http://schemas.xmlsoap.org/wsdl/">This returns a string value for Country.</wsdl:documentation>
+      <wsdl:documentation xmlns:wsdl="http://schemas.xmlsoap.org/wsdl/">This returns a string value</wsdl:documentation>

Notice that the change is for the documentation, so it is relatively harmless.

What this means is that you can now easily use your existing Splunk installation to monitor changes for your SOA configuration. The entire activity for monitoring SOA is part of SOA Governance. Keep in mind that just because you are monitoring file system changes does not in itself constitute an upward step in the SOA maturity model. However, with Splunk’s ability to monitor changes, instantly produce differences for what has changed, ability to provide active alerts for the changes, and ability to produce reports for analysis, you have a valuable tool in your arsenal for achieving a step closer towards SOA maturity. To achieve a more sophisticated and powerful approach towards monitoring your SOA configuration, consider also using the Splunk for Change Management Application, as it provides predefined reports and dashboards to facilitate change auditing, change detection, change reporting, change validation and incident response based on change events, change tickets and configuration files. I hope this article and example will get you closer for examining using Splunk for monitoring SOA configuration files.