DevOps Challenges

Builds are slow and the image is bloated; the quick fix on the table is buying more CI runners. Defend fixing the Dockerfile (multi-stage builds, layer ordering, smaller base images) instead of scaling hardware around the problem.

Nginx starts returning 502 immediately after a deploy and the reflex is to fiddle with the Nginx config. Defend investigating the upstream first — is the app process actually up, listening on the right socket/port, and healthy?

A contractor needs temporary database access and someone proposes opening the DB port to the internet. Defend secure alternatives: bastion/VPN access, time-boxed credentials, least privilege, and audit logging.

Long-lived AWS access keys are hardcoded on an EC2 instance and rotation is offered as the fix. Defend removing static keys entirely in favour of IAM instance roles and temporary credentials.

A deploy fails with permission errors and chmod 777 would make it go away. Defend diagnosing ownership and using correct users/groups and minimal permissions instead of opening everything to everyone.

Deployments are done by SSHing into the box and running git pull, late on Friday. Defend moving to a real deployment pipeline: repeatable builds, rollbacks, and deploy windows that don't ruin weekends.

The disk is full, the app is down, and deleting things at random is tempting. Defend finding what is actually consuming space (logs, caches, deleted-but-open files) before running anything destructive.

Someone wants to rewrite a messy history on the shared main branch with a force-push. Defend why rewriting shared history breaks everyone downstream and what the safe alternatives are.

Nightly backups run and everyone feels safe. Defend why backups that were never restored are not disaster readiness: restore testing, RPO/RTO, and off-site copies.

Tests pass locally but fail in CI, and hitting retry usually works. Defend treating flaky tests as real defects: environment differences, timing assumptions, and shared-state leaks.

A stale-cache incident makes someone propose --no-cache on every CI build. Defend fixing the caching strategy (layer ordering, cache keys) instead of throwing away Docker's build cache entirely.

A TLS certificate expires tonight and a long-lived self-signed cert would buy quiet. Defend proper automated renewal (ACME/Let's Encrypt), expiry monitoring, and why 10-year certs are a liability.

A t3 instance got slow and upsizing looks like the obvious cure. Defend checking CPU burst-credit depletion first — the metrics tell you whether you need a bigger box or a different instance family.

AWS keys landed in a public repository and a force-push to remove them feels like enough. Defend the real incident response: revoke immediately, audit for abuse, rotate everywhere, and assume the keys are compromised forever.

An S3-hosted site returns 403 and making the whole bucket public would fix it. Defend the safe options: CloudFront with origin access control, scoped bucket policies, and understanding what actually causes the 403.

Production is down at 3 AM after a deploy. Defend a position on rolling back versus fixing forward: what makes rollback safe, when forward fixes are justified, and how migrations complicate both.

A sudden traffic spike took the service down even though autoscaling was on. Defend why autoscaling alone can't absorb sharp spikes — scale-up lag, warm capacity, load shedding, and admission control.

A NOT NULL column has to land on a 50-million-row production table. Defend a zero-downtime migration plan: nullable-first, backfill in batches, validate, then enforce the constraint.

Every deploy is followed by 10 minutes of database pain, and a bigger instance is proposed. Defend fixing the cache stampede: cold caches after deploy, request coalescing, and cache warming.

Finance wants the whole fleet on Spot, database included. Defend the line between interruption-tolerant stateless workloads and stateful systems where a Spot reclaim means data loss or an outage.

Lambda concurrency is exhausting RDS connections and raising max_connections is on the table. Defend connection pooling (RDS Proxy) and concurrency control over cranking a limit that just moves the cliff.

On-call is flooded with alerts and mass-silencing is tempting. Defend redesigning the alerting: symptom-based alerts, severity tiers, and deleting alerts that never require action.

A pod cycles between Running and CrashLoopBackOff after a deploy and a teammate wants to raise CPU/memory. Defend a diagnostic-first approach: logs, exit codes, probes, and what actually changed in the release.

Pods keep getting OOMKilled and raising memory limits keeps buying time. Defend finding the leak: memory profiles, limits vs requests, and why blind scaling only postpones the same failure.

Manual console changes left Terraform plan full of drift. Defend a reconciliation strategy: decide per resource whether code or reality wins, then close the door on out-of-band changes.

SQS delivers the same message twice and the queue gets blamed. Defend at-least-once semantics: idempotent consumers and visibility-timeout tuning instead of switching queue technology.

A database was created by hand and Terraform wants to create its own. Defend importing the existing resource into state instead of letting Terraform destroy and recreate a live database.

A build container needs Docker, and mounting the socket with --privileged would work. Defend why that hands out root on the host, and argue for rootless builders (BuildKit, kaniko) or isolated build runners.

The fix is a one-line sysctl change and SSHing in takes a minute. Defend how to handle hotfixes when infrastructure is under configuration management — and how unmanaged edits silently rot.

The NAT Gateway bill exploded and adding gateways is proposed. Defend finding the traffic driver first: per-GB processing costs, VPC endpoints for S3/ECR, and cross-AZ data paths.

The pipeline is slow and skipping tests would ship faster. Defend making the pipeline fast instead: parallelization, test selection, caching — without giving up the safety net.

A queue backlog doubles daily and 10x-ing the consumers tonight sounds decisive. Defend diagnosing the bottleneck first — if consumers are blocked on a downstream, more of them makes it worse.

A 30-second dependency blip cascaded into a 2-hour outage, and more retries are proposed. Defend taming the retry storm: exponential backoff with jitter, retry budgets, and circuit breakers.

A production container is mining cryptocurrency. Defend a real incident response: isolate and preserve evidence, find the entry point, rotate what the workload could reach, and harden the pipeline that let it in.

An EKS pod needs AWS permissions and attaching them to the node role works instantly. Defend per-pod identity (IRSA / Pod Identity) over node-role permissions every pod on the node inherits.

Cross-account access to an encrypted bucket keeps failing and loosening policies is tempting. Defend solving the real chain: bucket policy, KMS key policy, and the caller's IAM policy must all agree.

DynamoDB throttles while table-level capacity looks idle. Defend the hot-partition diagnosis: skewed partition keys throttle locally no matter how much capacity you buy at the table level.

An async-replica failover lost 5 minutes of writes and leadership wants a zero-data-loss promise. Defend honest trade-offs: async vs sync replication, RPO you can actually commit to, and what sync costs in latency.

All infrastructure lives in one giant Terraform state. Defend splitting it: blast-radius reduction, faster plans, per-team ownership — and how to migrate state safely without downtime.

p99 latency spiked across a chain of microservices and every team blames another. Defend tracing the actual bottleneck with distributed tracing instead of guessing from per-service dashboards.

Automated secrets rotation broke production overnight and pinning long-lived credentials is proposed. Defend fixing the rotation mechanism — dual-secret overlap windows, graceful reload — instead of abandoning rotation.

Service A intermittently fails to reach Service B in the cluster. Defend a systematic diagnosis of the boring-but-hard middle: DNS resolution, conntrack, endpoints churn, and network policy.

Terraform plan shows the production database will be destroyed and recreated. Defend stopping the apply: understand what forces replacement, use lifecycle guards, and find a non-destructive path.

A server keeps OOM-killing processes and a huge swapfile would stop the kills. Defend finding what eats the memory: swap turns a fast failure into a slow, thrashing one.

A slow nightly report gets wrapped in one giant transaction. Defend why long-running transactions hurt Postgres — vacuum blocking, bloat, lock contention — and design the batch job properly.

An analysis needs per-user data and adding user_id as a Prometheus label would provide it. Defend cardinality discipline: unbounded labels blow up memory; use exemplars, logs, or traces for high-cardinality questions.

CI holds production credentials while pulling unpinned dependencies. Defend real supply-chain hardening: pinned and verified dependencies, OIDC short-lived deploy credentials, and least-privilege pipelines — beyond just adding a scanner.

The primary region is down and pressure mounts to fail over immediately. Defend a disciplined failover: confirm the outage scope, check replication state, follow the tested runbook, and know your split-brain risks.