Kubernetes Forensics: Secret Rotation from Ruin to Recovery

Background:

In this section, we will discuss specific actions during evidence collection, log analysis, and recovery for situations where it is necessary to determine if unencrypted secrets or sensitive variables without encryption have been used in Kubernetes.

NOTE: Skip the article if your are using encryption at a rest .

Out of box secret storing mechanisms in kubernetes:

Kubernetes has a built-in object/kind called a Secret. When you deploy a Secret, it uses the type "Opaque" by default, which stores generic key-value pairs. These values are stored in base64-encoded format, which can be easily decoded by a threat actor.

Example:

apiVersion: v1
kind: Secret
metadata:
     name: my-secret
type: Opaque
data:
    my-secret-var: VGhpcyBpcyBhIHNhbXBsZSBzZWNyZXQgdGV4dA==

Keep in mind that Kubernetes also allows secrets to be protected using additional mechanisms. However, in this article, our focus is on situations where these enhanced protections were not used and only basic secret encoding was applied.

Secrets Directory:

Secrets in Kubernetes are usually stored in etcd, which is a key-value database.

/var/lib/etcd/member/snap/db

Secret reveal:

As mentioned earlier, secrets are stored in base64 format, but having access to the etcd database file makes it much easier to retrieve and decode these secrets. Let's look into DB hex example:

As we know, base64-encoded values often end with ==. We can search for this pattern in the hex representation, which corresponds to the hex values 3D 3D 22. enter image description here

Writing a simple script can help reveal all stored plain passwords, making it easier to provide this information to the IT/Infrastructure team for revocation of those variables.

enter image description here

Conclusion

This method can be used when there is no running cluster, such as when the cluster has fallen due to an incident or has health issues. Of course, there are other methods to retrieve activity logs related to threat actor actions, but we will discuss those in future articles.