TL;DR: Intermittent failures can come from capacity, retries, races or inconsistent configuration across resolver paths. In a cluster the three usual causes are ndots:5 search-domain expansion increasing lookup traffic, CoreDNS being under-replicated or CPU-throttled, and conntrack races on parallel A and AAAA queries. Measure the query rate before you touch anything.
How to approach it
Say what intermittent tells you: some paths work; compare failing nodes, resolver configuration and query types before choosing load, timeout or race as the cause. Then give the three fingerprints. Interviewers ask this because it is common, subtle, and rewards someone who has actually read a resolv.conf.
A strong answer
Start by looking at the generated /etc/resolv.conf inside a pod, because it explains most of the volume:
search default.svc.cluster.local svc.cluster.local cluster.local ec2.internal
options ndots:5
ndots:5 means any name with fewer than five dots is treated as relative and tried against every search domain first. With this search list, an uncached api.example.com lookup can try four suffixes before the absolute name. Resolver behavior, early matches, caches and A/AAAA query handling determine the actual multiplier. At scale it is a multiplier on your entire DNS load, and it is the single highest-value thing to fix: set ndots:2 for pods that mostly talk externally, or use a fully qualified name with a trailing dot to skip the search path entirely.
Second, CoreDNS capacity. It is a normal deployment with normal limits, and a CPU limit on CoreDNS is a trap because throttling shows up as latency on every lookup in the cluster at once. Check its replica count against cluster size, check whether it is being throttled, and check its own metrics for a request rate that grew after a deploy. NodeLocal DNSCache is the standard structural fix: a per-node cache that removes most of the cross-node traffic and the conntrack pressure with it.
Third, the conntrack race. The classic Linux kernel behaviour where parallel A and AAAA queries from the same socket can collide in the NAT table and one gets dropped, surfacing as a 5-second timeout because that is the resolver's retry interval. A five-second spike is a retry-timeout clue, not proof of this race. Confirm the affected kernel/resolver versions and packet sequence. Mitigations are single-request-reopen in resolv.conf options, disabling IPv6 lookups if you do not use them, or NodeLocal DNSCache, which sidesteps the NAT path.
The measurement that separates these: CoreDNS request rate and latency, plus the ratio of NXDOMAIN responses. High NXDOMAIN can reflect search expansion or genuinely missing names. Rising latency needs CPU, upstream and network evidence. A five-second mode warrants checking retry timing and captures before attributing a kernel race.
What interviewers probe next
"Why five seconds specifically?" It is a common glibc resolver timeout before retry. Other resolvers and configurations differ; the interval identifies a timer worth checking, not its failure cause.
"Would you cache DNS in the application?" Carefully. It removes load but breaks the assumption that a Service IP change is picked up promptly, which matters during failover. Respect the TTL rather than caching forever.
"How would you prove it is DNS and not the network?" Resolve and connect separately in a test pod, timing each. If resolution is slow and connection to the resolved IP is fast, you have your answer.
Common mistakes
Adding CoreDNS replicas as the first move. If the cause is ndots expansion, you have scaled the symptom and left avoidable search queries in place.
Not knowing what ndots does, which is the most commonly asked follow-up in this whole area.
Treating a five-second spike as conclusive evidence without checking resolver timeouts and the packet path.
References