Incident Response in Kubernetes: Threat Hunting Techniques for Identifying Supply Chain Attacks
Background: If we are discussing one of the stages of incident response in Kubernetes—specifically log collection and evidence analysis—the approach is different from traditional methods used for on-premises environments or cloud VMs (such as EC2) and other systems.
Over the past year, we have seen a noticeable spike in supply chain attacks. As incident response specialists, it is important for us to be involved in forensic investigations—especially when the impacted asset is a Kubernetes cluster. By default, Kubernetes may not have any monitoring system in place, which makes forensic investigations more challenging. To effectively respond, we need to take proactive steps in preparation. Let’s discuss how to prepare for evidence collection and analysis in the context of supply chain attacks.
Architecture of evidence collection:
At this stage, we need to consider at a high level that some potentially risky actions are required:
- Setting up a DaemonSet
- Setting up a ConfigMap
- Setting up a ServiceAccount and assigning RBAC permissions to it
- Setting up StatefulSet volume (For persistance information)
The DaemonSet is used to schedule scanning pods on every node.
The ConfigMap is mounted as files into the container — the scanner code (scanner.py, entrypoint.sh) lives inside the ConfigMap. The scanner.py script calls the Kubernetes API to execute commands inside
other containers.
RBAC permissions are granted to the ServiceAccount that the scanner pod runs as — giving it visibility and exec access across the entire cluster.
StatefulSet volume is a single place all scanner pods ship their results to, backed by a persistent volume so data survives restarts.
As a result of this approach, you can obtain an inventory of all your assets and search for the necessary packages.
Conclusion: The approach described above is not a complete solution to the identified challenge; additional components should be incorporated during the incident preparation stage. There are also alternative methods available. However, this method can help you save time until a more comprehensive forensic analysis can be performed in your environment.