502 Bad Gateway Right After a Deploy
Scenario
You ship a routine release. Seconds later, users start hitting 502 Bad Gateway. The setup is a classic one: nginx sits in front as a reverse proxy and forwards traffic to your application listening on a local port. The deploy replaced the application build and restarted its service — and nginx itself was not touched at all. Still, because nginx is the thing printing the error, the immediate reaction in the incident channel is that nginx must be broken.
The Quick Fix on the Table
A colleague suggests going straight into the nginx configuration: adjust the proxy settings, maybe raise some timeouts, and run a reload. It feels productive — nginx is where the error appears, so nginx is where the fingers go.
The quick fix is on the table and the room is waiting for your call. Would you sign off on it? Take a position and justify it — out loud or on paper — before revealing the analysis.
Why the Quick Fix Fails
- 502 is nginx reporting, not nginx failing. A 502 means nginx tried to reach the upstream and got no usable response — the app is not listening, crashed on startup, or is refusing connections. Nginx is the messenger, not the culprit.
- The config did not change; the app did. The only variable in this incident is the deploy. Editing a config that worked five minutes ago introduces a second change on top of an unresolved first one.
- You risk masking the real bug. Bumping timeouts or fiddling with upstream blocks can turn a crisp "app is down" signal into slow, intermittent weirdness that is far harder to diagnose.
- You lose incident time. Every minute spent reloading nginx is a minute the actual failure — a crash loop, a missing dependency, a changed port — keeps the site down.
- Untracked config edits linger. Emergency nginx tweaks made under pressure rarely get reverted or reviewed, and they quietly become the next incident's confounding variable.
The interviewer nods: “Fine, the quick fix is off the table. So what exactly would you do — step by step?” Sketch your plan before revealing the approach.
The Right Approach
- Confirm the app process is actually up. Run
systemctl status app(or the equivalent for your process manager) and check for a crash loop or a failed start right after the deploy. - Read the app's own logs first.
journalctl -u app -n 100 --no-pageror the application log file will usually show the startup exception, missing environment variable, or migration failure directly. - Verify something is listening where nginx expects it.
ss -tlnp | grep 3000(or the configured port/socket) confirms whether the upstream endpoint exists at all — and whether the new build changed the port or socket path. - Test the upstream directly, bypassing nginx.
curl -i http://127.0.0.1:3000/healthfrom the same host tells you in one command whether the problem is the app or the proxy path. - Only then look at nginx's error log.
tail -f /var/log/nginx/error.logwill typically sayconnect() failed (111: Connection refused)— confirming the upstream diagnosis rather than pointing at nginx config. - Fix or roll back the deploy. If the new build cannot start, restore the previous release and let the site recover, then debug the broken build calmly outside the incident.
- Add a post-deploy health check. Make the pipeline curl the app's health endpoint before it declares the deploy successful, so the next bad build never reaches users as a 502.
Final pushback: “Your plan costs more time and money than the quick fix. Convince me.” How do you defend your position under pressure?
How to Defend It
- "502 means nginx couldn't get a response from the app. The error lives downstream of nginx, so that's where I look first."
- "Nothing in the nginx config changed — the deploy did. Debugging the thing that changed is faster than tuning the thing that didn't."
- "One curl against the local port settles the argument in ten seconds: if that fails, no nginx edit will fix it."
- "Changing proxy settings mid-incident adds a second variable. If we edit config now, we won't know whether the fix or the edit changed behavior."
- "Once we're stable, the real improvement is a health check in the deploy pipeline — then a broken build fails the deploy, not the users."