Kubernetes Incident Response Hunting: Identifying Anomalies in etcd During a Rapid Response Situation

Background: During incident response activities in Kubernetes, we need to ensure that artifacts have not been altered. One of the best places to hunt for artifacts is etcd, where Kubernetes stores all cluster state data and secrets.

What is ETCD: In simple terms, etcd is a key-value–oriented data store. It runs separately from the cluster but supports Kubernetes by storing and managing its important data. All data is Base64-encoded before being stored in the key-value store.

Storage access is governed by Role-Based Access Control (RBAC), limiting availability to privileged accounts and defined scopes.

The mod_revision value is incremented whenever the key's value is modified, serving as a change version marker.

While the mod_revision increments with every change, etcd (and by extension, Kubernetes) does not store old revision data indefinitely due to periodic compaction.

However, if you need to recover previous values, you can still access them via the command line—even after a change has occurred. Keep in mind that there is a limited opportunity window (typically 10–20 minutes) to retrieve this historical data before the compaction process permanently removes it.

etcdctl  get keyname --rev 4902(mod_revision value)  

If you know the current mod_revision but are unaware of the previous revision numbers, you can effectively brute-force the history. By using the current revision as your maximum value and decrementing the integer, you can query etcd for older states until you find the specific version of the data you need.

for rev in 3674 3722  3620  4902; do                                                                                                                                                            
  etcdctl  get username/2  --rev $rev
done

To perform a forensic analysis of etcd data, you can use the etcdctl command to extract the raw logs and state. By decoding the Base64-encoded values into human-readable plain text, you can reconstruct the system's history. This data serves as the foundation for building a forensic evidence and identifying unauthorized or accidental changes.

etcdctl get " " --prefix -w json | jq '.kvs[] | {                                                                                                                                                
  key_base64: .key,                  
  key: (.key | @base64d),
  value_base64: .value,
  value: (.value | @base64d),
  create_revision,
  mod_revision,
  version
}'

![]

Conclusion: In most environments, etcd is the 'Crown Jewel'—the primary target where both adversary and defender goals are most easily realized. Consequently, every Kubernetes environment must be hardened to ensure that this 'Holy Grail' of data remains inaccessible to threat actors, reducing the attack surface before an incident occurs.