Approaching Kubernetes Security — Detecting Kubernetes Scan with Splunk

The Kubernetes framework has become the leading orchestration platform. Originally developed by Google, Kubernetes is a "platform for automating deployment, scaling, and operations of application containers across clusters of hosts" * . The kubernetes platform is used in all Cloud platform provider vendors as a tool that allows orchestration, automation and provision of applications and specific needs computing clusters and services. 

Kubernetes drives the deployment of containers, which are small virtual machines that carry and operate stripped down operating systems, specific applications and a number of packages needed to run them. These containers are usually lightweight, interoperable, reusable, and provide security by isolation and segmentation if deployed correctly.

Kubernetes framework allows quick deployment of resources in a streamline fashion and with a preset architecture that allows scaling resources on a needed basis.

A kubernetes cluster is basically composed of a Kubernetes Master and Nodes. A Kubernetes Master is a collection of three processes that run on a single node in your cluster, which is designated as the master node. Those processes are: kube-apiserver, kube-controller-manager and kube-scheduler.*  A special object within the cluster master is Etcd which stores the entire estate of the cluster, its configuration, specifications and the statuses of the running workloads. 

The non master nodes in a cluster run two processes. The Kubelet, which is basically a node agent that communicates with master node and Kube Proxy, which is the network proxy used by the cluster components to communicate. Other components of the cluster are CAdvisor which is an open source resource usage collector (Web Interface), and of course Kube-Nodes which are the worker machines in the cluster.

The nodes are basically virtual or physical machines that run the applications. The kubernetes architecture is also composed of several abstractions defined as objects, these objects are important to understand the functioning of a Kubernetes cluster and of course its attack surface. These objects are described as:

• Pod: A Pod is the basic execution unit of a Kubernetes application–the smallest and simplest unit in the Kubernetes object model that you create or deploy. A Pod represents processes running on your cluster*
• Service: An abstract way to expose an application running on a set of Pods as a network service
• Volume: is just a directory, possibly with some data in it, which is accessible to the Containers in a Pod.
• Namespace: “Intended for use in environments with many users spread across multiple teams, or projects.*
   

There are also other abstractions for functionality references within the kubernetes architecture such as Deployment, DaemonSet, StatefulSet, ReplicaSet and Job. These objects are specifically designed to provide specific configuration and function items within a Kubernetes cluster.  There are some Kubernetes objects that deserve special attention because of their value from a security perspective. Those objects are ConfigMaps and Secrets. Let’s look at why these 2 objects are important to look for after to keep clusters safe.

Configmaps: Includes things such as bind configuration files, command line arguments, environment variables, port numbers and other system components are runtime. This object is used for configuration replication and reference and should be stored in a central and protected place. Think for example of a hardcoded API key that is used by multiple applications distributed in containers.

Secrets: Stores and manages sensitive information, such as passwords, Oauth tokens and ssh keys. Unauthorized access to these files may result in cluster compromise on a multiple scale.

How Do You Access Kubernetes Clusters?

The kubernetes framework is available from the main 3 cloud providers Amazon AWS EKS, Google GKE and Microsoft AKS. Every vendor has its own version and every vendor's version does have its own features but in general they all pretty much refer to the same architecture. It is important to note that Kubernetes can be also deployed on premises, although this post refers mainly to cloud based deployments. 

There are mainly two ways of accessing a kubernetes cluster.  One via the API using Kubectl client libraries or by making REST requests. These requests go through  several stages which include authentication, authorization and Admission control.  It is important to understand that from the cloud provider’s perspective, these processes are already integrated into their IAM offerings. The following are some of the relevant remote access items in Kubernetes clusters.

• API transport security port: 6443
• Authentication: Client Certificates, Password, Plain Tokens, Bootstrap tokens and JWT Tokens.
• Authorization: ABAC, RBAC  or Webhooks (Depends on provider setup)
• Admission control: Software modules that can modify or reject requests
• API Server Ports and IP addresses ports: 8080, 6443, 443
   

Other ports to watch on a Kubernetes clusters include *

As seen on the information above Kubernetes clusters attack vectors can be divided in two areas:

From Inside The Cluster:

• Application Vulnerabilities
• Container Implantation
• CI/CD Devops exposure/compromise
   

From Outside The Cluster:

• Exposed API
• Exposed Kubelet
• Information Disclosure
• Exposed Management GUI
• Denial of Service
   

Inside The Cluster

Application vulnerabilities can lead to cluster compromise because they are running within the environment outside control of k8s. They are a result of misconfiguration or running with high privileges. Container Implantation can lead to compromise of the entire cluster because they “run” as with the permission of the pod/node/cluster. if an implanted container gets deployed, the attacker is inside your environment. Splunk security research team has provided previous research in regards of the exposure and risks facing CI/CD Devops workflows, where components such as source code repositories that are supposed to catch library vulnerabilities and bugs/exposures using Jenkins or CircleCI pipelines which incorporate security tests.  However if those vulnerabilities are not fixed, they show up in production. A simple example is a hardcoded API key left in a code repository like github or a container deployed with high privileges that allows attackers to read Secrets objects.

Outside The Cluster

Although there might be an intermix of outside and inside attack vectors, Kubernetes clusters can be attacked from the outside mainly when they must support an exposed application. Also there are several recorded instances of misconfiguration and vulnerabilities that have been compromised from remote attackers accessing the cluster directly, specifically things like access to Kubelet, ETCD, API . 

We can search directly and get more detailed information using ESCU stories. In the following examples we can see the latest release from Splunk ESCU that includes Kubernetes Scan Detection stories.

Getting Started Detecting Kubernetes Scan

Because Kuberentes framework is so widely used and it is becoming the main engine for application and service deployments in cloud environments it is important to monitor and detect if malicious actors are probing around exposed services and applications. The first step in any remote attack is footprinting, which consists of a series of actions that allows the attacker to discover operating systems, protocols, applications and services and subsequently probe for vulnerabilities and exploits. 

This Splunk ESCU release contains an Analytic story called Kubernetes Scanning Activity that identifies attacker footprinting of a kubernetes environment.  It includes detection searches for Amazon EKS Kubernetes cluster, Amazon EKS Kubernetes Pod and GCP Kubernetes cluster.

To develop this scenario we have a kubernetes cluster that is exposed to the internet.  We analyzed the access to learn specific information about a remote attacker scanning the cluster. The goal for detecting and investigating reconnaissance is to know who and where it’s coming from. The theory is the attacker doesn’t know your environment, so activities that are “probing in nature”, like success/failure ratios, user agent anomalies are indicative of attack scans to find or verify your environment. Therefore, you want to learn:

• What is the source IP 
• What is the user agent indicative of a browser, command line tool or web service
• What is the HTTP verb used in request 
• What are response codes from these requests (Allowed, Forbidden, Not found)
• What is the destination of these requests
   

In the next graph we can see unauthenticated requests from internet IP addresses, with a verb “get”, a responseStatus.code 403 (Forbidden), with unusual user agents (checks, go http client,  HTTP banner detection) and requests URI that clearly indicate looking for files or even command execution, indicating the intention to probe and detect the destination resources. This is the nature of a probing, detection scan.

The Kubernetes Scan story in this release includes searches for both AWS and GCP. 

This release also includes a detection for AWS EKS Pod scanning. In the Pod scanning search we can obtain granular information on specific requests targeting Pods API (/api/v1/pods). In the following example we can see the following.

• What is the source IP: Not visible in the graphic but present in search. 
• What is the user agent indicative of a browser, command line tool or web service (curl/7.58.0, python requests/2.21.0, Mozilla 5.0…)
• What are response codes from these requests (403, Forbidden)
• What is the destination of these requests  (/api/v1/pods)
   

For more searches and information on Kubernetes Scanning activity story please download Splunk ESCU release. 

^{_{About Splunk Security Research Team
The Security Research Team is devoted to delivering actionable intelligence to Splunk's customers, in an unceasing effort to safeguard them against modern enterprise risks. Composed of elite researchers, engineers, and consultants who have served in both public and private sector organizations, this innovative team of digital defenders monitors emerging cybercrime trends and techniques, then translates them into practical analytics that Splunk users can operationalize within their environments. Download Splunk Enterprise Security Content Update in Splunkbase to learn more.}}