Access Keys Are Hardcoded on the EC2 Box. Just Rotate Them?
Scenario
An application on EC2 talks to AWS services using long-lived access keys sitting in a config file on the instance. The keys belong to an old IAM user with broad permissions, they have never been rotated, and the instance still allows IMDSv1. Now one of those key pairs has been compromised, and the team needs to respond.
The Quick Fix on the Table
A teammate proposes the minimal loop: deactivate the leaked pair, mint a new access key for the same IAM user, paste it into the same config file, and call the incident closed. Ten minutes, no architecture changes, everything keeps working exactly as before.
The quick fix is on the table and the room is waiting for your call. Would you sign off on it? Take a position and justify it — out loud or on paper — before revealing the analysis.
Why the Quick Fix Fails
- It reloads the same gun. The root cause is a long-lived, broadly-permissioned secret stored in plaintext on a server. A fresh key in the same file has the same lifetime, the same blast radius, and the same leak paths — backups, AMI snapshots, log lines, shell history, anyone with SSH.
- It skips the incident response. A compromised key with broad permissions may already have been used. Rotating without checking what the key did leaves persistence mechanisms — new users, new keys, modified roles — sitting in the account.
- Broad permissions stay broad. The old IAM user can likely do far more than the application needs. The next leak is just as catastrophic as this one.
- IMDSv1 stays open. The instance metadata service in v1 mode is exposed to SSRF-style credential theft, so even the "fixed" instance has a known credential-leak vector left running.
- Manual rotation doesn't happen. These keys went unrotated for years. Nothing about this fix changes that — the new key will also live forever, and the same conversation happens after the next leak.
The interviewer nods: “Fine, the quick fix is off the table. So what exactly would you do — step by step?” Sketch your plan before revealing the approach.
The Right Approach
- Kill the leaked credential immediately. Deactivate the compromised pair now —
aws iam update-access-key --status Inactive— and attach a deny-all inline policy to the user while you investigate. Containment first. - Audit what the key actually did. Search CloudTrail for the exposed key ID: look for unfamiliar API calls, new IAM users or keys, role changes, resource creation in unusual regions. Treat anything suspicious as an active compromise, not a config chore.
- Create an IAM role scoped to what the app needs. Write a least-privilege policy from the application's real API usage (IAM Access Analyzer and CloudTrail help here) instead of inheriting the old user's broad permissions.
- Attach the role to the instance.
aws ec2 associate-iam-instance-profilegives the box automatically rotated temporary credentials via the metadata service — no secret on disk, nothing for a human to rotate, nothing to leak from a config file. - Point the application at the default credential chain. Every AWS SDK and the CLI pick up instance-role credentials automatically once the hardcoded keys are removed from the config file and environment.
- Enforce IMDSv2.
aws ec2 modify-instance-metadata-options --http-tokens requiredcloses the SSRF credential-theft path that IMDSv1 leaves open. - Retire the IAM user and prevent recurrence. Delete the user's remaining keys, then add detection: an AWS Config rule or credential report check that flags long-lived access keys, and secret scanning in CI so static keys never land in files or repos again.
Final pushback: “Your plan costs more time and money than the quick fix. Convince me.” How do you defend your position under pressure?
How to Defend It
- "Rotation swaps the secret but keeps the vulnerability. The problem isn't which key is in that file — it's that any long-lived key is in a file at all."
- "A compromised key with broad permissions is an incident, not a chore. Before we close anything, CloudTrail has to tell us what that key did."
- "An instance role gives the app credentials that rotate themselves every few hours and never touch disk. That's strictly less work than manual rotation, forever."
- "These keys sat unrotated for years — that's evidence our process won't maintain static keys. The fix that survives our own habits is the one with no static keys."
- "The role migration is an afternoon: attach the profile, delete two lines of config, enforce IMDSv2. One afternoon versus repeating this incident annually is an easy trade."