Kubernetes Incident Response Hunting: Identifying Malicious Traffic at the Node Level

Background:

Sometimes, you may encounter situations where no logs are being stored for your Kubernetes pods. In such cases, you still need to investigate potential malicious network activity using Indicators of Compromise (IOCs). In this article, we’ll explore the available options and the practical steps you can take to analyze network traffic directly at the node level. To hunt for network IOCs, follow these simple steps:

• Install tcpdump on the Kubernetes node
• Identify the pod’s internal network interface

Steps: Get Node IP

kubectl get nodes -o wide

SSH into NODE

ssh username@yourip

Get POD names

kubectl get pod -A

Get POD running port

kubectl get pod NameOfThePod -o wide

Now use tcpdump to look for the outgoing traffic and IOC

tcpdump -i any host PODINTERNALIP -nn -s 0 -X   

This steps will help at least on incident active stage find and correlate IOC .

Background: This is not the only way to reveal outgoing connections. However, different measurement approaches have shown that, in some scenarios within Kubernetes, it may not be possible to capture all outgoing connections from within pods—because different pods can be deployed with different types of containers and network configurations.