Secrets Rotation Took Down Prod at 3 AM. Pin Credentials?
Scenario
Shortly after 3 AM, a newly automated rotation job changed the production database password: it updated the database and wrote the new value to the secrets manager, exactly as designed. What it did not account for was every service that had already read the old value — connection pools and cached configs across a dozen services kept presenting the retired credential, authentication failures cascaded, and production was down for an hour and a half. The rollout had every classic mistake at once: first production run applied fleet-wide, executed overnight, and no alerting on authentication-failure rates. The rotation itself only existed because a security audit required it.
In the blameless-in-name-only postmortem, a colleague draws the hard line: "Automation just proved it can take us down. Pin the credentials, rotate by hand once a year, and we never have this outage again."
The Quick Fix on the Table
Disable automated rotation and pin long-lived credentials. The outage class disappears immediately, nobody gets paged at 3 AM by a rotation job, and the change costs nothing today.
The quick fix is on the table and the room is waiting for your call. Would you sign off on it? Take a position and justify it — out loud or on paper — before revealing the analysis.
Why the Quick Fix Fails
- It swaps a loud failure for a silent one. A rotation outage pages you, gets fixed, and gets a postmortem. A leaked long-lived credential works quietly for an attacker for months. Pinning trades a visible availability risk for an invisible security risk — and the invisible one is the expensive kind.
- Long-lived secrets accumulate exposure. Every laptop, CI log, crash dump, backup, and departed employee that ever touched the credential remains a live threat for as long as the credential is valid. Rotation is what puts an expiry date on all of those leaks.
- Annual manual rotation is the same outage on a slower fuse. If services cannot survive a credential change today, they still can't next year — except by then the procedure is forgotten, the person who did it left, and the change is even scarier. Rarely exercised operations are the least safe ones.
- It fails the audit that mandated rotation. The requirement came from security governance. Quietly reversing it buys a finding in the next audit and re-opens the whole discussion with less credibility.
- The outage wasn't caused by rotation — it was caused by a bad rollout. Fleet-wide first run, no overlap window, no reload mechanism, no auth-failure monitoring, off-hours timing. Blaming "rotation" for that is like blaming deployments because someone shipped untested code to every region at once.
The interviewer nods: “Fine, the quick fix is off the table. So what exactly would you do — step by step?” Sketch your plan before revealing the approach.
The Right Approach
- Adopt the dual-secret (two-key) pattern. Never have a moment where only the new credential is valid while clients hold the old one. Create the new credential alongside the old (a second DB user, or the built-in alternating-users strategy in AWS Secrets Manager rotation lambdas), let both authenticate during an overlap window, and revoke the old one only after telemetry shows nothing is using it.
- Make services reload secrets gracefully. Clients should fetch credentials at connection time (not process start), respect the secrets manager's version stages (
AWSCURRENT/AWSPREVIOUS), and treat an auth failure as "re-fetch the secret and retry" before alerting. Connection pools need a recycle path that drains old connections instead of clinging to boot-time values. - Instrument the failure mode you just met. Alert on database authentication-failure rate and on "connections still using the previous credential" before any revocation step. The rotation job itself should verify the new credential works (open a test connection) before promoting it, and halt with an alarm otherwise.
- Roll rotation out like you roll code out. First in staging, then one low-criticality production service, then progressively wider — during business hours, with an owner watching the dashboards and a documented, tested rollback (re-promote the previous secret version).
- Rehearse revocation as a drill. Run a game day: rotate, watch every service pick up the new value, revoke the old, confirm zero auth failures. When that drill is boring, rotation frequency can safely increase.
- Reduce the number of secrets that exist at all. Where the platform allows it, replace static credentials with short-lived issued ones — IAM database authentication, workload identity, or dynamic secrets from Vault-style engines. A credential that expires in 15 minutes barely needs "rotation" as a concept.
Final pushback: “Your plan costs more time and money than the quick fix. Convince me.” How do you defend your position under pressure?
How to Defend It
- "The outage wasn't rotation working — it was rotation done without an overlap window, without reload support, without monitoring, fleet-wide on the first run, at 3 AM. Every one of those is fixable; the leak risk of pinned credentials isn't."
- "A rotation failure pages us and is over in ninety minutes. A leaked pinned credential works for an attacker silently until we find out from someone else. I'll take the failure mode that wakes us up."
- "Annual manual rotation is the same event with all the practice removed. If our services can't survive a credential change, that's the bug — and it doesn't fix itself by doing the change less often."
- "Dual-secret rotation makes the outage structurally impossible: old and new are both valid during the window, and we revoke only after telemetry says nobody uses the old one. That's a mechanism, not a promise to be careful."
- "The audit required rotation for a reason. Let's come back to security with a hardened rotation design and a drill schedule, not with a request to waive the control that just embarrassed us."