A Container in Prod Is Mining Crypto

Why the Quick Fix Fails

• It shreds the evidence. The running processes, the miner binary, network connections, shell histories, and any dropped tooling live in the pod's ephemeral filesystem and memory. Delete the pod and the forensic record of how the attacker got in is gone — permanently.
• The entry point is still open. Whatever let the attacker in — a vulnerable dependency in the public-facing service, a bad image, an exposed endpoint — is untouched. The ReplicaSet will even schedule a fresh pod from the same image behind the same service. Expect a new miner within days, and an attacker who now knows to be quieter.
• The credentials must be presumed stolen. The pod held a service account token and readable secrets. An attacker with code execution had both. Deleting the pod does not un-steal them — those credentials work from anywhere until they're rotated, and the miner may have been the distraction while they were exfiltrated.
• The blast radius is unassessed. With a service account token, the attacker may have talked to the Kubernetes API, touched other workloads, or moved laterally. "Is one pod compromised, or is the cluster?" is the question that decides the whole response — and pod deletion answers it with a shrug.
• It miscategorizes the event. This is a security breach with possible customer-data implications, not a resource anomaly. Treating it as a CPU problem skips disclosure assessment, stakeholder notification, and the post-incident hardening that stops recurrence.

The Right Approach

1. Isolate — don't terminate. Cut the pod off while keeping it alive for forensics: apply a deny-all NetworkPolicy scoped to the pod, remove it from the Service by relabeling (which also stops the ReplicaSet controller from counting it), and kubectl cordon the node if node-level compromise is plausible. The mining stops earning; the evidence stays warm.
2. Preserve evidence before anything changes. Capture kubectl logs --previous and current logs, kubectl get pod -o yaml, process and connection listings from the node's runtime (crictl inspect, ps, ss via node access — avoid execing through the compromised container), a filesystem snapshot or dd image of the container layer, and the image digest actually running. Timestamp everything and store it off-cluster.
3. Rotate every credential the pod could reach — assume theft. Delete and reissue the service account token, rotate all mounted/readable Secrets and the external credentials they represent (database passwords, API keys, cloud keys), and audit cloud-provider credentials reachable from the node (IMDS). This is the single most time-critical step after isolation.
4. Hunt for lateral movement. Review Kubernetes audit logs for API calls from that service account, cloud audit trails (e.g. CloudTrail) for use of reachable credentials, and network logs for the unknown host — then search the whole cluster for other connections to it. Decide on evidence, not hope, whether the incident is one pod or many.
5. Find and close the entry point. Correlate the compromise time with ingress access logs on the public-facing service, scan the image and its dependencies for known RCEs, and check for exposed debug endpoints or leaked credentials. Do not redeploy the same artifact unchanged — that's re-arming the trap.
6. Eradicate and redeploy hardened. Only after evidence and rotation: delete the pod, rebuild the image patched, and redeploy with a tightened posture — automountServiceAccountToken: false unless required, least-privilege RBAC, runAsNonRoot and a read-only root filesystem, default-deny egress NetworkPolicies (a miner that can't reach its pool is a very quiet failure), and runtime detection (e.g. Falco) so the next anomaly is caught by tooling rather than a CPU graph.
7. Run the formal post-incident process. Write the timeline, assess data exposure and any disclosure obligations with legal/compliance, and turn the findings into pipeline controls: image scanning gates, admission policies, and an incident runbook so the next responder doesn't have to improvise at 2 a.m.

How to Defend It

• "Deleting the pod treats this as a CPU problem. It's a breach: someone executed code in our production environment, next to a service account token and our secrets. The miner is the symptom they chose to be loud about."
• "The pod is the crime scene. Everything that tells us how they got in lives in its filesystem and memory — delete it and we're rebuilding the investigation from guesswork."
• "Kill the pod and the ReplicaSet redeploys the same image behind the same public service with the same hole. We're not removing the attacker; we're refreshing their instance."
• "Isolation gets us everything deletion promises — the mining stops within minutes — while keeping the evidence. We lose nothing by doing this properly except the illusion of being done."
• "Every credential that pod could read is compromised until rotated. That's true whether the pod lives or dies, and it's the clock we should actually be racing."