TL;DR: 500 is your application throwing; 502 means the proxy got an invalid or refused response from upstream; 503 means nothing was available to send it to; 504 means upstream accepted it and did not answer in time. The last three are emitted by the proxy, which is why the application log can be silent.
How to approach it
Say who emits the code first, because that is the diagnostic value. A 500 comes from your application and will be in its logs. A 502, 503 or 504 usually comes from the load balancer or ingress, and the application may have logged nothing at all, which is the thing that confuses people into believing the logs are broken.
A strong answer
500 Internal Server Error. Your application ran and threw. An unhandled exception, a null dereference, a failed database call with no error handling. The stack trace is in your logs, the request reached your code, and this is the easiest of the four because the evidence exists where you expect it. Check the application log for that request, ideally by trace ID.
502 Bad Gateway. A proxy tried to reach your application and got something it could not use: connection refused, connection reset, a malformed response, or the process died mid-response. Common causes are the application not listening on the port the proxy expects, the container crashing on startup so nothing is behind the service, a worker process being killed while handling the request (often the OOM killer), or a protocol mismatch such as the proxy speaking HTTPS to a plaintext listener. In Kubernetes a 502 frequently means the pod is running but the container is not actually serving yet.
503 Service Unavailable. The proxy had nothing to send the request to, or deliberately refused it. In Kubernetes this is the classic "no ready endpoints" case: pods exist, readiness probes are failing, so the Service's endpoint list is empty. It also comes from a proxy's own queue being full, from a circuit breaker opening, or from rate limiting. The distinguishing feature from 502 is that nothing was attempted upstream.
504 Gateway Timeout. Upstream accepted the connection and did not respond within the proxy's timeout. This is the one that points at your application being slow rather than broken: a query with no index, a downstream call with no timeout of its own, a thread pool exhausted so the request queued. The number to check immediately is the proxy's timeout against your application's own p99, because a 30-second proxy timeout in front of a p99 of 28 seconds produces intermittent 504s that look random.
The practical sequence:
# is it every request or some?
kubectl logs -n prod deploy/ingress-nginx --tail=100 | grep ' 50[234] '
# does the Service have anything behind it?
kubectl get endpoints -n prod checkout
# is the app itself logging the request at all?
kubectl logs -n prod deploy/checkout --since=10m | grep <trace-id>
If the ingress logs show the code and the application logs show nothing, the request never arrived, which narrows it to 502 or 503 territory and puts the problem between the proxy and the pod.
Two others worth knowing because they get confused with these. 499 in nginx logs means the client gave up and closed the connection first, which usually means you are slower than the caller's timeout rather than failing. And 429 is rate limiting, which is a deliberate refusal rather than a fault.
What interviewers probe next
"502 or 503, which worries you more?" 503 with no ready endpoints means the whole service is down. A 502 is often a subset of pods. Check the proportion of requests affected before deciding which is worse.
"Why would 504s appear only at peak?" Queueing. Utilisation rises, requests wait for a worker, total time crosses the proxy timeout. The application may be processing at the same speed per request; the waiting is what changed.
"How do you find which hop is slow?" Compare the proxy's upstream response time with the application's own request duration. If the proxy sees 8 seconds and the application reports 200 milliseconds, the time is in connection setup, queueing or the network between them.
Common mistakes
Treating all 5xx codes as one condition, which discards the information about which hop failed.
Searching the application log after a 503 and concluding logging is broken, when the request never reached the application.
Raising the proxy timeout to stop 504s, which hides a slow path and moves the failure to the client.
Ignoring readiness probes as a cause of 503, which is the most common explanation in Kubernetes.