Shadow Workloads: How to Find and Monitor Unchecked Static Pods on Your Nodes
Background: During incident response activities at the cluster level, we should not focus solely on pods created via kubectl that are managed by the Kubernetes control plane (kubelet and etcd). It is important to recognize that a threat actor can deploy pods directly at the node level—such as via static pod manifests—without involving the API server or etcd. These pods may remain completely outside of API visibility and management, making them difficult to detect and eradicate using standard Kubernetes tools.
Attacker flow for static pod deployment: To be able to deploy a static pod to a node, an attacker has a limited set of options. The most common method is to gain direct access to the node—typically via SSH or another remote management interface—and place a YAML file in the static pod manifest directory (e.g., /etc/kubernetes/manifests). However attacker can leverage other methods, too
- Taking advantage over vulnerbilites like RCE
- Use stolen credential for privileged user
Defender flow to spot static pod deployment: First of all, as IR specialists, we need to collect information about the existence of files inside /etc/kubernetes/manifest or /var/lib/kubelet/.
In this attack simulation we can see unknown yaml file in manifest
Keep in mind that the location of the manifest folder can be changed by an attacker via the kubelet using the --pod-manifest-path flag. Example: kubelet --pod-manifest-path=/etc/myDummpyPath
Another method to detect unknown static pods is to implement a general log analytics rule in your SIEM. When a threat actor creates a static pod on a node, the system typically leaves traces in the logs.
Defender flow to eradicate spotted static pod deployment: Because static pods are not managed by the Kubernetes API server, using kubectl commands to remove them is ineffective. Even if you delete the pod, the system will automatically recreate it as long as the static pod manifest file exists. Example
kubectl delete pod podName -n Namespace
And to remove it you need to navigate to file in manifest and remove it over RM command .
Conclusion: To prevent/detect such kind of attack flow just follow to simple steps 1. Monitor any SSH access to any Node in your cluster 2. Monitor static pod creation over SIEM 3. Monitor leaked credential related to kuberntes users 4. Keep up to dated your environment 5. In case of pod remove always remember that you can not eradicate it over kubectl (API), instead of that search file and remove it from manifest .