The NAT Gateway Bill Exploded. Add More Gateways?
Scenario
The monthly AWS bill takes a jump, and Cost Explorer points a finger: NAT Gateway data processing is now the single biggest line item. The architecture is common — workloads in private subnets reach the outside world through one NAT gateway, and "outside" includes pulling container images from ECR and moving objects to and from S3. There are no VPC endpoints in the account. Someone glances at the gateway's throughput graphs, decides it looks "saturated," and the conversation turns to capacity.
The Quick Fix on the Table
Add a second NAT gateway (and maybe a third) to spread the load. It frames the bill as a bandwidth problem: the pipe is full, so add pipes. It's a quick Terraform change and nobody has to figure out where the traffic is actually coming from.
The quick fix is on the table and the room is waiting for your call. Would you sign off on it? Take a position and justify it — out loud or on paper — before revealing the analysis.
Why the Quick Fix Fails
- NAT gateways bill per gigabyte, not per gateway-hour alone. The exploding line item is data processing — a charge on every GB that transits the gateway. Two gateways processing the same total traffic cost the same in processing fees, plus an extra hourly charge. The "fix" strictly increases the bill.
- "Saturated" was never diagnosed. A single NAT gateway scales to tens of gigabits on its own. Without flow-log evidence, the throughput graph shows that traffic is high, not that the gateway is the bottleneck — and high traffic is exactly the thing that costs money.
- The biggest traffic sources probably shouldn't touch NAT at all. ECR image pulls and S3 object traffic from private subnets flow through the NAT gateway by default — and both have VPC endpoint alternatives. Paying NAT processing fees to reach other AWS services in the same region is the classic silent money leak.
- It buries the real question. Something changed to make the bill jump — a new deployment loop pulling multi-GB images, a chatty batch job, a misbehaving retry. Adding capacity institutionalizes the waste instead of finding it.
The interviewer nods: “Fine, the quick fix is off the table. So what exactly would you do — step by step?” Sketch your plan before revealing the approach.
The Right Approach
- Get the evidence first. Enable VPC Flow Logs on the NAT gateway's ENI (or query existing logs with
Athena/ CloudWatch Logs Insights) and rank traffic by destination and by source. In an hour you'll know exactly which talkers and which destinations account for the gigabytes. - Add the free win immediately: an S3 gateway endpoint. A gateway-type VPC endpoint for S3 (
com.amazonaws.<region>.s3) costs nothing, needs only a route-table entry, and pulls every byte of S3 traffic out of the NAT path. If S3 is a top destination, this alone may fix the bill. - Add interface endpoints for ECR and friends. Interface endpoints for
com.amazonaws.<region>.ecr.api,ecr.dkr, plus CloudWatch Logs, STS, Secrets Manager — whatever the flow logs show. These have an hourly + per-GB price, so do the arithmetic: for image-pull-heavy clusters they're usually far cheaper than NAT processing. - Hunt the anomaly. Compare flow-log volume against the date the bill jumped. Common culprits: a CI/CD loop pulling huge images on every deploy, missing image caching on nodes, a data job downloading from a public bucket, or a retry storm against an external API. Fix the behavior, not just the routing.
- Check cross-AZ paths. If workloads in one AZ route through a NAT gateway in another, you're paying cross-AZ data charges on top of processing. Per-AZ gateways are justified for this and for availability — which is a different, defensible reason than "the bill is high."
- Then right-size deliberately. After endpoints and fixes land, re-read Cost Explorer. If genuine internet-bound traffic still needs more capacity or AZ redundancy, add gateways as an informed availability decision — and set a billing alarm plus a NAT-bytes dashboard so the next jump is caught in days, not at invoice time.
Final pushback: “Your plan costs more time and money than the quick fix. Convince me.” How do you defend your position under pressure?
How to Defend It
- "NAT charges by the gigabyte processed. Splitting the same gigabytes across two gateways changes nothing except adding a second hourly fee — this fix makes the bill larger by construction."
- "We haven't actually diagnosed saturation; we've observed expensive traffic. Flow logs will tell us in an hour who's talking and to what — let's spend the hour before we spend the money."
- "If a chunk of this is S3, the fix is literally free: a gateway endpoint and a route-table entry, and that traffic never touches NAT again."
- "ECR pulls through NAT means we pay processing fees every time a node pulls an image. Interface endpoints for ECR cost a fraction of that at our volume — that's arithmetic, not opinion."
- "The bill jumped, which means something changed. If we scale capacity now we're making the waste permanent and invisible; I'd rather find the change and then size capacity for the traffic that's actually legitimate."