Upgrading Linux Forwarders Using the Deployment Server

As a Splunk Consultant, I commonly interact with customers using older or mixed versions of Splunk's Universal Forwarder in their environment. This usually happens when the Splunk admin relies on a different team to push updates, does not have ssh access to those servers, or the customer simply does not have a universal way of maintaining packages across their environment. Splunk provides a convenient way of maintaining Splunk configuration files and apps across thousands of servers, but no direct support of pushing out installer upgrades using the Deployment Server. I asked myself, could we use the Deployment Server to upgrade forwarders without ssh access? We can, by using a scripted input and a double fork.

A double fork is commonly used by almost all Linux services to create a daemon. A daemon is a process that runs in the background instead of being under direct control of a user. When a Splunk process runs a scripted input, the script becomes a child process of splunkd. If Splunk is told to stop (a requirement of upgrading), the scripted input will also stop. By using a double fork, the scripted input becomes disowned by the splunkd process and becomes a child of init or systemd, depending on your operating system. Both init and systemd are the parent of all process on the system. They are executed by the kernel and are responsible for starting all the other processes.

Let me first demonstrate this concept using a basic example. I created a script called whoami.sh with the following contents.

#!/bin/bash
while true; do whoami; done

It was executed in a typical manner.

[root@uf01 ~]# ./whoami.sh

You can see by running pstree that the whoami.sh script is a child process of bash.

[root@uf01 ~]# pstree
systemd─┬─NetworkManager─┬─dhclient
	│                └─2*[{NetworkManager}]
	├─agetty
	├─auditd───{auditd}
	├─chronyd
	├─crond
	├─dbus-daemon───{dbus-daemon}
	├─lvmetad
	├─master─┬─pickup
	│        └─qmgr
	├─polkitd───5*[{polkitd}]
	├─rsyslogd───2*[{rsyslogd}]
	├─splunkd─┬─splunkd
	│         └─34*[{splunkd}]
	├─sshd─┬─sshd───bash───whoami.sh───whoami
	│      ├─sshd───bash
	│      └─sshd───bash───pstree
	├─systemd-journal
	├─systemd-logind
	├─systemd-udevd
	├─tuned───4*[{tuned}]
	└─vmtoolsd───{vmtoolsd}

I also ran the same script using a scripted input and you can see that it is a child process of splunkd.

[root@uf01 ~]# pstree
systemd─┬─NetworkManager─┬─dhclient
	│                └─2*[{NetworkManager}]
	├─agetty
	├─anacron───run-parts─┬─awk
	│                     └─yum-cron
	├─auditd───{auditd}
	├─chronyd
	├─crond
	├─dbus-daemon───{dbus-daemon}
	├─lvmetad
	├─master─┬─pickup
	│        └─qmgr
	├─polkitd───5*[{polkitd}]
	├─rsyslogd───2*[{rsyslogd}]
	├─splunkd─┬─splunkd───whoami.sh───whoami
	│         └─34*[{splunkd}]
	├─sshd─┬─3*[sshd───bash]
	│      └─sshd───bash───pstree
	├─systemd-journal
	├─systemd-logind
	├─systemd-udevd
	├─tuned───4*[{tuned}]
	└─vmtoolsd───{vmtoolsd}

I then created a wrapper script which calls whoami.sh.

#!/bin/bash
( /opt/splunkforwarder/etc/apps/whoami/bin/whoami.sh & )

As you can see, the whoami.sh script is now a child of systemd, even though it was executed by splunkd.

[root@uf01 ~]# pstree
systemd─┬─NetworkManager─┬─dhclient
	│                └─2*[{NetworkManager}]
	├─agetty
	├─auditd───{auditd}
	├─chronyd
	├─crond
	├─dbus-daemon───{dbus-daemon}
	├─lvmetad
	├─master─┬─pickup
	│        └─qmgr
	├─polkitd───5*[{polkitd}]
	├─rsyslogd───2*[{rsyslogd}]
	├─splunkd─┬─splunkd
	│         └─33*[{splunkd}]
	├─sshd─┬─sshd───bash───pstree
	│      └─sshd───bash
	├─systemd-journal
	├─systemd-logind
	├─systemd-udevd
	├─tuned───4*[{tuned}]
	├─vmtoolsd───{vmtoolsd}
	└─whoami.sh───whoami

Using the same technique as above, I created a very basic upgrade.sh script that stops Splunk, extracts the tarball, and restarts Splunk. I also included the tarball inside the app that is deployed using the Deployment Server. This prevents the need of having all of the forwarders download the upgrade package. This script will only run if the installed version of Splunk is different than the version set in the script. This script makes several assumptions such as the install location, installer type, and system architecture.

#!/bin/bash
# set splunk path
SPLUNK_HOME=/opt/splunkforwarder
# set desired version
NVER=6.5.2
# determine current version
CVER=`cat $SPLUNK_HOME/etc/splunk.version | grep VERSION | cut -d= -f2`
if [ "$NVER" != "$CVER" ]
then
    echo "Upgrading Splunk to $NVER."
    $SPLUNK_HOME/bin/splunk stop
    tar -xvf $SPLUNK_HOME/etc/apps/upgrade_linux_uf/static/splunkforwarder-6.5.2-67571ef4b87d-Linux-x86_64.tgz -C /opt
    $SPLUNK_HOME/bin/splunk start --accept-license --answer-yes
fi

The app layout.

├── upgrade_linux_uf
│   ├── bin
│   │   ├── upgrade.sh
│   │   └── wrapper.sh
│   ├── default
│   ├── local
│   │   ├── app.conf
│   │   └── inputs.conf
│   └── static
│       └── splunkforwarder-6.5.2-67571ef4b87d-Linux-x86_64.tgz

Upgrading using a tarball will work regardless of the permissions granted to the splunkd process since you are only writing to the $SPLUNK_HOME directory. If you are going to use this methodology to upgrade a .rpm or .deb, you will need to make sure that the splunkd user has proper permissions.

Once you have deployed this app using the Deployment Server you can watch the forwarder versions change by monitoring the _internal index or by using the REST api on the Deployment Server to check the build number of the remote forwarders.

| rest /services/deployment/server/clients

You can view the complete app on Github. Use at your own risk.