Approaching Azure Kubernetes Security

The Splunk Security Research Team has been working on Kubernetes security analytic stories mainly focused on AWS and GCP cloud platforms. The turn has come now for some Azure Kubernetes security monitoring analytic stories. As outlined in my "Approaching Kubernetes Security — Detecting Kubernetes Scan with Splunk" blog post, when looking at Kubernetes security, there are certain items within a cluster that must be monitored.

Let’s quickly recap the fundamentals of a Kubernetes cluster. It is composed of a Kubernetes Master and Nodes. A Kubernetes Master is a collection of three processes that run on a single node in your cluster, which is designated as the master node. Those processes are: kube-apiserver, kube-controller-manager and kube-scheduler.* A special object within the cluster master is Etcd which stores the entire estate of the cluster, its configuration, specifications and the statuses of the running workloads. 

The none master nodes in a cluster run two processes. The Kubelet, is a node agent that communicates with master node and Kube Proxy, the network proxy used by the cluster components to communicate. Other components of the cluster are CAdvisor which is an open source resource usage collector (Web Interface), and of course Kube-Nodes which are the worker machines in the cluster.

Composed of several abstractions defined as objects, they are important to understand when looking at the functioning of a Kubernetes cluster and of course its attack surface. 

These objects are described as:

• Pod: A Pod is the basic execution unit of a Kubernetes application–the smallest and simplest unit in the Kubernetes object model that you create or deploy. A Pod represents processes running on your cluster*
• Service: An abstract way to expose an application running on a set of Pods as a network service
• Volume: is just a directory, possibly with some data in it, which is accessible to the Containers in a Pod.
• Namespace: “Intended for use in environments with many users spread across multiple teams, or projects.*
   

There are also other abstractions for functionality references within the kubernetes architecture such as Deployment, DaemonSet, StatefulSet, ReplicaSet and Job. These objects are specifically designed to provide specific configuration and function items within a Kubernetes cluster.  

There are some Kubernetes objects that deserve special attention because of their value from a security perspective. Those objects are ConfigMaps and Secrets. Let’s look at why these 2 objects are important to look for after to keep clusters safe.

• Configmaps: Includes things such as bind configuration files, command line arguments, environment variables, port numbers and other system components are runtime. This object is used for configuration replication and reference and should be stored in a central and protected place. Think for example of a hardcoded API key that is used by multiple applications distributed in containers.
  
• Secrets: Stores and manages sensitive information, such as passwords, Oauth tokens and ssh keys. Unauthorized access to these files may result in cluster compromise on a multiple scale.
   

From the access perspective when looking at a Kubernetes cluster it is important to look at specific ports for Kubernetes objects and services. Specifically those that can expose the Kubernetes API, the Kubelet or any other component to external unauthorized access and possibly allow attackers to send malicious data. 

Azure Kubernetes Cluster Security Monitoring

After this quick recap let's take a look at the new detection searches and analytic stories for Azure Kubernetes. First let's look at Kubernetes Azure Scan fingerprint detection. This detection aims to provide information on possible fingerprinting scans from the internet. The main fields used for this detection are: source ip address, user agent, http verb, requested URI and response status. In the following screenshot we can see for example user agent strings for Zgrab , a scanning tool popular for discovering and fingerprinting hosts on the internet. 

In some cases banners for popular vulnerability scanner Nmap can be clearly observed.

Kubernetes monitoring can be challenging from within and from outside of clusters, this search should give cloud administrators or secops teams a quick way to spot suspicious requests coming from the internet which are forbidden in many cases by default.

Kubernetes Azure Pod Scan Fingerprint Detection

We can delve even deeper into details by looking at the Pod level. As we can see in the following detection search result. In this case using similar fields such as source IP address, user agent, http verb, request URI and Pod request information, it can be clearly observed an user agent from scanning tool Nmap and the distinct verbs targeting the POD and with specific request URIs that indicate probing for files or directories.

Objects and Roles That Need to be Monitored

In this new release of Azure Kubernetes analytic stories, we have searches that focus on access to internal sensitive objects (Configmaps, Secrets) and Roles with high privileges (Clusterrole, Clusterrolebinding). As outlined in this blog these objects and roles are deemed sensitive because they have either information that can help attackers to compromise the cluster or roles that would allow attackers to escalate privileges. Both new analytic stories are in the context of Kubernetes RBAC, however they are distinctly focused. One is focused on the sensitive objects being accessed and the other is focused on what role is used to access.

Kubernetes Sensitive Object Access Activity

This analytic story focuses on accounts accessing Kubernetes cluster sensitive objects such as configmaps or secrets providing information on items such as user, usergroup. object, namespace and authorization reason.  Below is a detection search that provides Kubernetes Azure detection of access to sensitive object access within the context of the user group, accessed sensitive object, namespace and RBAC reason.

While the above search may provide a broad view into sensitive role usage, it is important to understand, that access to such objects must be monitored and the provided search can as well be modified for specific field values such as:

• user.username=system.anonymous . Anonymous users accessing sensitive objects such as configmaps and secrets is something that needs to be addressed by any security personnel monitoring a Kubernetes cluster. This would mean unauthenticated accounts accessing sensitive objects within a cluster.
• annotations.authorization.k8s.io/decision!=allow . This field shows if request access has been allowed or not. Within the security context of a Kubernetes cluster with pre-established access policies, a high number of denied requests, specially towards sensitive objects may indicate attackers trying to access sensitive information contained wihtin these sensitive objects.  
   

This analytic story is also complemented with two more searches that provide information such as source IP of requesting client, Kubectl calls and accounts with forbidden or failure status.

Kubernetes Sensitive Role Activity 

This analytic story focuses on detection and response around sensitive role usage within a Kubernetes cluster against cluster resources and namespaces. The information provided includes source IPs, user name, activity by pod, response status, namespace, and http verbs. 

In the following example we can see a breakdown of source IP addresses with usernames, with different verbs (get, update) with failure status against specific pods. Using the number of these requests, plus their source IPs and statuses we can determine if there are suspicious IP addresses with failed requests to specific Pods. This can be indicative of something that should be investigated further by the analyst monitoring a kubernetes cluster.

This search provides as well a contextual view of role usage against specific pods and namespaces. Namespaces are very important when monitoring Kubernetes clusters as they are used as units of separation and organization of resources. Within this context there are three fields to consider when using this detection search:

• sourceIPs{} . Are these IPs whitelisted or blacklisted? . Is there such a policy when accessing a Kubernetes cluster?, even more importantly if the IP in case is unknown it is important to further investigate the reason of association with a sensitive role as shown in the other fields and specific resources. 
• user.usergroups{} . This field provides information on the user authentication group to which related users belong. This has to be observed within the context of source of user request (IP Address) and targeted resource and role usage. This within a context where RBAC is applied to Kubernetes access policies. 
• annotations.authorization.k8s.io/reason . This field provides the RBAC information context from user, user group and targeted resource (requestURI) within a namespace. 
   

This story is also complemented with two more searches that provide information on Kubernetes Azure detect RBAC authorization by account, and Kubernetes Azure detect sensitive role access.

_{About the Splunk Security Research Team}

_{The Security Research Team is devoted to delivering actionable intelligence to Splunk's customers, in an unceasing effort to safeguard them against modern enterprise risks. Composed of elite researchers, engineers, and consultants who have served in both public and private sector organizations, this innovative team of digital defenders monitors emerging cybercrime trends and techniques, then translates them into practical analytics that Splunk users can operationalize within their environments. Download Splunk Enterprise Security Content Update in Splunkbase to learn more.}