TL;DR: A 504 comes from a gateway that gave up waiting, so it is emitted by a component your backend metrics do not describe. The request either never reached the backend or reached it and the response did not return in time. Look at the hops between: queueing at the ingress, connection pool exhaustion, NAT port capacity, DNS, and the difference between server-side latency and time in queue.
How to approach it
Point out immediately that backend metrics measure requests the backend accepted, so a request that never got accepted is invisible to them. That reframing is the answer; everything after is where to look.
A strong answer
Your service dashboard reports duration for requests it handled. If the request sat in an accept queue for nine seconds and then completed in 40 milliseconds, your dashboard shows 40 milliseconds and the user saw a timeout. That systematic blindness is why this question is hard and why "the backend is fine" is usually a true and misleading statement.
Work the hops in order.
The gateway's own view. The proxy or load balancer records both upstream response time and total time. Check the proxy's exact field definitions: high total and low upstream time can indicate client upload, downstream transmission, queueing or other overhead. Connection setup may already be included in upstream timing. This one comparison resolves a large share of these incidents.
Accept queue and worker saturation. If every worker or thread is busy, new connections queue in the kernel and wait. The application never sees them, so latency looks fine, and netstat -s overflow counters or the runtime's own queue depth is the evidence. This is the most common cause when the pattern is intermittent and correlates with traffic.
Connection pool exhaustion. Between the gateway and the service, or between the service and the database. A pool sized for normal load runs out under a spike and callers block waiting for a connection. Time is spent waiting for a pool slot, which almost nobody instruments, so it appears as unexplained latency at the caller and nothing at all downstream.
NAT and conntrack. In cloud environments a NAT gateway has a port allocation per destination and a conntrack table with a size limit. Exhaust either and new connections are dropped silently, which presents as intermittent timeouts under load with every service healthy. This is a favourite because nothing in the application layer indicates it.
DNS. Resolution happens before the connection, so its latency is invisible to both ends. An intermittent five-second stall can be a resolver retry timeout, and it produces exactly this symptom if the client timeout is above it.
Timeout mismatch. If the gateway waits 10 seconds and the backend's own timeout to the database is 30, the gateway gives up first and the backend keeps working on a request nobody is waiting for, consuming a worker. Timeouts should decrease as you go inward, and getting this backwards converts a slow dependency into a cascading failure.
The measurement that settles it: instrument client-perceived latency, either from the edge or from a synthetic probe running the full path. Until you measure the whole path, you are comparing one segment against a complaint about the journey.
What interviewers probe next
"Difference between 502, 503 and 504?" HTTP defines 502 as an invalid upstream response, 503 as temporary inability to serve, and 504 as failure to receive a timely upstream response. Proxies map concrete failures differently: a 504 can include connection establishment timeout, and a 503 does not uniquely mean no healthy backends. Read the emitting proxy's reason code.
"How would you test the connection pool theory?" Measure acquisition wait, active connections and backend saturation first. A bounded canary pool increase can test the hypothesis if the backend has headroom, but can also move or worsen the bottleneck.
"Why intermittent rather than constant?" Almost every cause here is a capacity limit, and limits are only reached at peak.
Common mistakes
Accepting "the backend is fine" as evidence. It measures accepted requests only.
Not comparing upstream time to total time at the proxy, which is the fastest available discriminator.
Raising timeouts to make the error disappear, which converts fast failures into slow ones and hides the cause.
References