Getting Started with Citrix in Splunk - [Part 1]

With most of the world on lockdown due to the COVID-19 virus, many aspects of IT services and digital transformation have been put into the fast lane. There are reports of massive surges in the use of tools such as Zoom, Microsoft Office 365, etc. in order to communicate and collaborate. At the same time organizations are required to scale up access to their internal applications. We automatically think of technologies like VPN to provide network connectivity to corporate networks from a remote location such as our home offices.

The digital workspace platform Citrix has been particularly popular, not only for its ability to deliver applications and content within internal networks but also to securely and efficiently deliver apps and content to remote devices/locations. Recently I have seen organizations massively scale up capacity of their Citrix environments in order to accommodate an increase in their user base due to remote access requirements to company resources via Citrix. With an increasing need to work remotely and to fast track the transformation of how we work, these services and technologies become business-critical, hence an increased importance for monitoring and observing them.

Questions? Answers!

How do you start onboarding data from your Citrix environment? How can Splunk as the Data-to-Everything Platform assist you? You may need answers to questions such as: What should I look at? Why are my logons taking so long? Which applications are being run by our users? What’s the end-user experience? How much capacity do I need to accommodate the additional load?.

Typical Citrix XenApp/XenDesktop deployment

In a Citrix XenDesktop™ 7 or XenApp™ 7 environment there are multiple parts that are equally important, which only make up a complete solution when working properly with each other. A typical Citrix deployment consists of the following points as a minimum, also shown in the Citrix architecture diagram hereunder:

• Users or Devices
• NetScaler
• Controller
• Database
• License Server
• Virtual Desktop Agents

The below diagram indicates where you would use Splunk to pick up relevant data via a Universal Forwarder (pink icon) or in the case of Citrix NetScaler™, we will utilize Syslog in the first step. The scenario here is an on-premise/private cloud deployment.

From here on, I will guide you through the sequential steps on how to add relevant data into Splunk. These steps are aligned with the logical flow showing how users typically connect, as shown in the diagram above. Furthermore, I will give some examples of ready-to-use apps and add-ons which will enable you to acquire value even faster while allowing you to dive into more advanced topics and areas.

Starting with the ‘Users’ layer

You may or may not experience a mixture of managed devices as well as BYOD (Bring Your Own Device). In order to collect data from your clients’ devices, you could install managed end-user devices with a Splunk Universal Forwarder.

There is Never Not a NetScaler

It is only on rare occasions that Citrix installations are not accessed via a NetScaler - be it as a load balancer or residing in the DMZ acting as a remote access gateway. In order to collect data from the NetScaler, data can be sent via Syslog, IPFIX, and the NITRO API. Splunk Add-on for Citrix NetScaler was created to ingest data in a CIM-compliant format to support apps such as Splunk Enterprise Security (ES) and Splunk IT Service Intelligence (ITSI). It also includes prebuilt panels to visualize the data. Installation and configuration steps are documented in Splunk docs.

Splunk App for Infrastructure and/or Windows Add-on

When we take a look back at our diagram, you’ll notice that the part we haven’t yet covered is the one that talks mainly about Windows Servers, for which we are going to use the Universal Forwarder with a combination of add-ons. There are two options:

• Use the Splunk App for Infrastructure to ‘easy install’ the Universal Forwarder preconfigured to ingest key Windows metrics as well as Windows EventLogs via a Powershell script 
• Install the Splunk Add-on for Microsoft Windows on top of your Universal Forwarders and enable the data sources you want to collect

The advantage of using the Splunk App for Infrastructure is that it comes with a lot of Splunk goodness to easily monitor and troubleshoot your entire infrastructure. Moreover, it combines logs and metrics in one place.

For the Citrix StoreFront™ component, this approach of ingesting the Windows data is mostly sufficient. Most Storefront functionality is based on Windows Internet Information Services (IIS) and you can also use the Splunk Add-on for Microsoft IIS to ingest IIS information for more low-level use cases.

Who is in Control?

The ‘Control’ layer houses the Controller - the brain of everything - as well as a license server and database server(s). The Citrix licensing mechanism has a very generous grace period that makes it fairly robust against license server failures. If it should ever fail, it is easy to restore the license service. The Windows monitoring as described above is sufficient for it. The Database in a Citrix environment is the memory of the Citrix brain, the controller and it plays an important role in the entire picture. For these reasons, I recommend that you monitor it with the Splunk App for Infrastructure or the Windows Add-on. In addition to the Windows monitoring, there is also a Splunk Add-on for Microsoft SQL Server to collect more detailed metrics and logs about the database server.

Now, let’s get to our brain - the Citrix Controller. It typically resides on a Windows host. In addition to collecting Windows data, it is also important to gather additional Citrix-specific data. In order to do so, I recommend making use of add-ons and an app created by fellow Splunker Jason Conger. One add-on has been created for the controllers (brokers), the other for the Citrix Virtual Delivery Agents (VDA), both of which are hosted on GitHub. Install the TA-XD7-Broker Add-on on the Universal Forwarder on your controllers and enable the data sources in the configuration files that you want to ingest into Splunk. 

A similar procedure is also required in the ‘Resources’ layer where your VDAs reside, but instead of the TA-XD7-Broker, deploy the TA-XD7-VDA Add-on. In the mentioned GitHub repository there is a detailed Installation manual giving you a step by step guide on how to install and configure the add-ons as well as the template app for XenDesktop. 

The template app will provide some dashboard panels to address several use cases such as ICA latency reporting, user logon time details, application usage, critical service monitoring, etc. It is called "template app" as it is meant to be customized for your environment. As with all things Splunk, there are many options to adapt, expand, and to go above and beyond.

Going the Extra Mile

You may notice that both add-ons also use PowerShell scripts to collect data from Citrix components. The Universal Forwarder then sends the output of these scripts to Splunk. Hence, there is the possibility to customize as much as you want with your own PowerShell scripts. As you may know, you are able to get any piece of information about your Citrix environment via SDKs and APIs.

The other layers of ‘Hosts’ and ‘Management’ can also be integrated into Splunk to give you a complete picture of your entire stack from end-to-end. Given that ‘Hosts’ could be anything from bare metal to hypervisors or private or public cloud, this would go beyond the scope of this blog post. Have a browse on our app store to find add-ons and apps for other shared Infrastructure of your IT environment on splunkbase.com and discover for yourself that Splunk is, in fact, the Data-to-Everything Platform.

uberAgent for an OOB Experience

On Splunkbase you will discover that users in the Splunk community, as well as technology partners, publish their own apps and add-ons. An app that I would like to mention is the uberAgent from “vast limits GmbH”. In the context of Citrix XenApp/XenDesktop, the uberAgent helps you collect all relevant metrics and information. It also helps visualize it with their Splunk App, giving you an out-of-the-box experience and addressing the most important use cases in monitoring Citrix environments. Vast limits recently announced on their blog that they are going to provide 2 months of uberAgent for unlimited users for FREE to monitor #WFH. Visit their blog and website for more detailed information about the uberAgent and what else it can do beyond Citrix XenApp/XenDesktop monitoring. :)

In case you were wondering: Yes, there is going to be part 2 which will cover more of the Citrix Cloud topics, including hybrid environments.

So stay tuned and happy Splunking!

Christian

----------------------------------------------------
Thanks!
Christian Radeke