The t3 Instance Got Slow. Just Size It Up?
Scenario
A web application runs on a t3.medium. Mornings are fine, but during busy hours the service visibly slows and response times climb. The CPU graph has a distinctive shape: utilization rises with traffic, then flattens against a hard ceiling well below 100%, as if the instance is hitting an invisible wall. Nobody has looked at the CPU credit metrics yet.
The Quick Fix on the Table
A colleague suggests the intuitive move: the box is clearly too small, so upgrade to a t3.large or t3.xlarge. More CPU, problem gone, ten-minute change.
The quick fix is on the table and the room is waiting for your call. Would you sign off on it? Take a position and justify it — out loud or on paper — before revealing the analysis.
Why the Quick Fix Fails
- The graph shape says throttling, not saturation. A machine that is genuinely out of CPU pegs at 100%. A curve that flattens against a hard line below 100% is the signature of t-family burst credits running out — the instance is being throttled to its baseline, not maxed out.
- A bigger t3 keeps the same failure mode. Every t3 size earns credits at a fixed rate and throttles when they run dry. A t3.large just has a larger tank; sustained load above baseline still drains it, and the same slowdown returns at a slightly higher traffic level — after you've doubled the bill.
- It's a diagnosis-free spend. Without looking at the credit metrics, "size up" is a guess. If the guess is wrong, you've spent money, changed a variable, and still have a slow app — plus less credibility for the next fix.
- It can mask a software problem. If credits are being drained by a runaway worker, a hot loop, or missing caching, throwing hardware at it hides the defect until it outgrows the next instance too.
- Wrong-family spending compounds. Burstable instances are priced for spiky, mostly-idle workloads. Paying for progressively bigger burstable instances to run a sustained load is the most expensive way to buy steady CPU.
The interviewer nods: “Fine, the quick fix is off the table. So what exactly would you do — step by step?” Sketch your plan before revealing the approach.
The Right Approach
- Confirm the diagnosis in CloudWatch. Plot
CPUCreditBalanceandCPUCreditUsageagainstCPUUtilizationfor the slow periods. A balance hitting zero exactly when the ceiling appears is the smoking gun; five minutes, zero dollars. - Check whether unlimited mode is saving or surcharging you. Look at
CPUSurplusCreditBalanceand the instance's credit specification (aws ec2 describe-instance-credit-specifications). Standard mode explains throttling; unlimited mode explains a quiet extra line on the bill instead. - Measure the true CPU need. Average utilization over a week tells you the workload's steady demand. Compare it to the t3.medium baseline (20%): occasionally above means burstable is fine; persistently above means the workload has outgrown the burstable model, not the size.
- Stop the bleeding cheaply if needed. Switching the existing instance to unlimited mode (
aws ec2 modify-instance-credit-specification) removes the throttle immediately for a bounded per-vCPU-hour charge — a sensible bridge while you decide the real fix. - Move sustained load to a fixed-performance family. If demand is genuinely constant, an m-series (general) or c-series (compute-heavy) instance delivers 100% of its CPU all day with no credit arithmetic — often at a similar price to the larger t3 being proposed.
- Alert on the leading indicator. A CloudWatch alarm on
CPUCreditBalancebelow a threshold warns you before the throttle engages, instead of your users doing it during peak hours. - Profile before you conclude. While metrics are open, check what consumes the CPU — if a bug or missing cache is draining credits, fixing it may make the whole resizing debate moot.
Final pushback: “Your plan costs more time and money than the quick fix. Convince me.” How do you defend your position under pressure?
How to Defend It
- "CPU flat-lining below 100% isn't a small instance — it's a throttled one. Real saturation pegs at 100; this is the burst-credit ceiling."
- "Five minutes with CPUCreditBalance in CloudWatch confirms or kills the theory for free. Resizing first is spending money to avoid reading a graph."
- "A bigger t3 has the same physics: earn credits, spend credits, throttle at baseline. We'd pay double to move the wall, not remove it."
- "If the workload is sustained, the right answer is a fixed-performance family — an m or c instance gives all of its CPU all day, often for what the bigger t3 costs."
- "If we're wrong about the cause, sizing up buries the evidence. Diagnose, then spend — the fix we can defend on the graph is the one that sticks."