An EKS Pod Needs AWS Access. Attach It to the Node Role?
Scenario
A workload running in your EKS cluster needs to read objects from an S3 bucket. Right now it has no AWS credentials, so every call fails with AccessDenied and the feature is blocked. The cluster's worker nodes run under an instance role that carries the standard node permissions — ECR pulls, CNI networking, and so on — and the cluster already has an OIDC provider configured. The team wants the pod unblocked today.
The Quick Fix on the Table
Attach an S3 read policy to the node instance role. The AWS SDK inside the pod will pick up credentials from the instance metadata service automatically — no manifests to change, no new roles to create, working code in five minutes.
The quick fix is on the table and the room is waiting for your call. Would you sign off on it? Take a position and justify it — out loud or on paper — before revealing the analysis.
Why the Quick Fix Fails
- It grants S3 to every pod on every node, forever. The instance role is ambient: any container scheduled onto those nodes — today's workloads, next quarter's, a third-party ingress controller, a batch job from another team — inherits the same S3 access through the metadata endpoint. One pod asked; the whole cluster received.
- It turns any pod compromise into an S3 breach. If any container on the node is exploited — not the one that needed access, any of them — the attacker curls the instance metadata service and walks away with credentials that read your bucket. The blast radius of every future incident just grew to include this data.
- Node roles only ratchet upward. The next team repeats the trick for DynamoDB, then SQS, then KMS. Within a year the node role is a super-role nobody can inventory, and removing any permission risks breaking an unknown consumer — so nobody ever does. The quick fix compounds into permanent over-privilege.
- You lose attribution entirely. Every S3 call from the cluster appears in CloudTrail as the node role. When something misbehaves — or an auditor asks who can read that bucket — "some pod, on some node" is the only answer available.
- The right mechanism is already installed. The cluster has an OIDC provider, which is the one prerequisite for IRSA. The team would be choosing the insecure path while the secure one sits configured and waiting.
The interviewer nods: “Fine, the quick fix is off the table. So what exactly would you do — step by step?” Sketch your plan before revealing the approach.
The Right Approach
- Create a purpose-built IAM role with least privilege. Scope the policy to exactly what the workload does:
s3:GetObject(ands3:ListBucketif needed) on the specific bucket ARN — nots3:*, notResource: "*". - Write the trust policy for one service account. The role trusts the cluster's OIDC provider, with a condition pinning the
subclaim tosystem:serviceaccount:<namespace>:<serviceaccount>and theaudclaim tosts.amazonaws.com. Only tokens minted for that exact service account can assume it. - Bind it in Kubernetes. Create the dedicated service account annotated with
eks.amazonaws.com/role-arn: <role-arn>, and setserviceAccountNamein the pod spec. EKS's webhook injects a projected, auto-rotating web identity token; the AWS SDK exchanges it viaAssumeRoleWithWebIdentitywith no code changes. - Or use EKS Pod Identity if you prefer the newer path. Install the
eks-pod-identity-agentadd-on and create an association withaws eks create-pod-identity-associationmapping namespace + service account to the role. Same per-pod outcome, simpler trust policies, easier to manage at scale across many clusters. - Verify identity, not just success. From the pod, run
aws sts get-caller-identity— it should return the new role, not the node role — then confirm the S3 read works and the CloudTrail entry shows the workload's own role with the service account in the session context. - Block the ambient fallback. Enforce IMDSv2 with a hop limit of 1 on the node group so pods can't silently fall back to node credentials, and audit the node role down to what nodes themselves require (ECR, CNI, CloudWatch agent). Node identity for nodes; workload identity for workloads.
- Template it for the next team. Capture the role + service account pairing as a Terraform module or Helm values block, so the five-minute path in this codebase is the secure path and the node-role shortcut never looks attractive again.
Final pushback: “Your plan costs more time and money than the quick fix. Convince me.” How do you defend your position under pressure?
How to Defend It
- "Putting S3 on the node role doesn't give access to our pod — it gives it to every container that ever lands on those nodes. That's not a permission grant, it's a cluster-wide default."
- "After this change, compromising any pod on the node is enough to read that bucket via the metadata endpoint. We'd be wiring our data's blast radius to the weakest workload in the cluster."
- "IRSA is maybe an hour of work here — the OIDC provider already exists. We're comparing sixty minutes against a permanent hole that an auditor will eventually bill us far more to unwind."
- "With a per-workload role, CloudTrail tells us exactly which service touched the bucket. With the node role, every investigation starts with 'it was one of the nodes' — which is no answer at all."
- "Node roles only grow — next month it's DynamoDB, then KMS, and nobody will dare remove anything. Per-pod identity keeps every permission attached to the one workload that justified it."