Google SRE / Production Engineer interview questions

The question that separates people who have debugged a hung rollout from people who have read about one. The interesting half is not the signal, it is PID 1.

A staple of every networking round, and the follow-up is always about TLS termination. Knowing which layer sees what decides most of the answer.

The oldest question in systems interviewing and still the best, because it is unbounded: the interviewer stops you when you run out of depth, and where you stop is the score.

The question that reveals whether someone knows what load average actually counts on Linux. It is not a CPU metric, and that single fact resolves the whole thing.

The classic ephemeral port exhaustion question. Most candidates know the phrase TIME_WAIT; far fewer know which side accumulates it and why the popular fix is dangerous.

The scripting round in disguise. The constraint is the question: it exists to see whether you stream or whether you load, and everything else is about whether the script is safe to run twice.

The two-syscall duet behind every process on the box. The follow-ups live in what survives the transition, and candidates who skip that fail the next question.

Everything top, htop and your APM agent display is derived from procfs anyway. Knowing the raw files turns any bare SSH session into a monitoring console.

Multiplexing removes application-level queuing and hands the problem to TCP. Here is where the blocking really lives, the loss math behind it, and the four levers that help.

Loss-based versus model-based congestion avoidance decides whether your WAN transfers fill the pipe or die on the first stray drop. Here is the operator's decision, not the RFC summary.

Anycast turns BGP convergence into your failover mechanism. Sessions stick to the site they landed on, monitoring must go per-site, and a route you cannot withdraw cleanly is worse than DNS.

Spreading instances across availability zones is not the same as surviving an AZ failure. The candidates who know this can name the four coupling points that quietly re-unify your blast radius.

A cluster boundary does not always create a new cloud identity. Inspect the workload pool and principal identifier before assuming a leaked key.

The Terraform question that actually filters. Everyone knows both meta-arguments exist; the answer is about how state addresses resources, and why an index is a fragile key.

The single most common Kubernetes interview question, and the one where a vague answer is most obvious. Every stage you name is a place the interviewer can ask you to debug.

The pair everyone configures and few can explain. QoS hints at eviction risk, but usage versus requests and Pod priority determine node-pressure ranking, and CPU and memory behave completely differently when you exceed them.

The question that finds out whether you know Kubernetes networking or just use it. The model has three rules and everything about CNI follows from them.

PDB governs voluntary disruptions only and does not gate rolling updates at all. Candidates who say 'it keeps N pods alive' have just described the part it does not do.

Init containers are the easy half. The hard half is why plain sidecars race your app at both ends of the pod's life, and what restartPolicy Always changes about that.

A registry stores repositories, which hold manifests, which point at content-addressed blobs. Candidates who can draw that hierarchy and the auth dance around it understand every pull failure they will ever meet.

The kubelet is a local reconcile loop with its own event stream, probe workers and eviction brain. Staff-level candidates can narrate it and explain why Ready status can reflect either reported health failure or missing heartbeats.

Every control plane component depends on the API server, so who starts the API server? The bootstrap paradox sits at the heart of every cluster, and the kubelet plus a directory of files is the accepted way out.

SSH plus yum is how outages start. The senior answer replaces nodes from versioned images on a cadence, orchestrates with surge capacity and PDBs, and can explain the kernel-skew problem that makes fleet homogeneity a feature.

Upgrades look like rolling restarts right up to the morning the CNI stops talking to the new kubelet. The scored answer is ordering, skew policy, and addon sequencing, not courage under pressure.

The classic five-second stall is rarely CoreDNS being slow; it is UDP, conntrack races and ndots expansion. The diagnosis ladder and the NodeLocal DNSCache fix separate operators from readers of architecture blogs.

Three API servers behind one health-checked endpoint, etcd on its own quorum, leader election everywhere else. The scored part is naming what degrades quietly before anything alerts.

Kubernetes does not see GPUs, it sees an extended resource that something has to advertise. When that advertisement is missing or wrong, the scheduler is behaving correctly and the node is lying.

A Job with a sidecar was broken by design for years: the workload finishes, the proxy does not, and the pod never terminates. Native sidecars fix it by making the container an init container that keeps running.

Changing pod resources historically required recreation, which made vertical autoscaling disruptive for workloads with expensive recovery. In-place resize changes that, with limits worth knowing.

Ingress solved routing by convention and annotation, which meant every controller invented its own. Gateway API's real contribution is not features, it is splitting one object into three so that platform and application teams stop editing the same file.

An end-of-maintenance date changes the risk even while traffic stays healthy. The work is proving routing and security parity before moving a production hostname.

Clusters multiply for good reasons: regions, isolation, tenancy, blast radius. The problem is not creating them, it is that cluster seven diverges from cluster three and nobody knows until an upgrade behaves differently.

Route attachment and cross-namespace backend access have different owners. Use route status to find the failing boundary before broadening any permission.

A question where both answers are defensible and the reasoning is the entire score. The operational consequences are concrete: what happens when a shared library needs a security patch.

Long-lived branches pile up merge debt until integration becomes a gamble. There is a named technique that ships a three-month refactor in releasable increments, and its final step is deletion.

The honest mechanics of merging to trunk many times a day: flags, fast CI, trunk locks, and the conditions under which you should not run it at all.

Build graphs identify affected targets only when their dependencies are complete. The ladder from build-everything to content-addressed caching, hermeticity audits, and where undeclared inputs rot correctness.

Ownership files decay faster than org charts. What separates working CODEOWNERS from rubber-stamp theatre is hygiene: team aliases, stale-owner linting, and keeping review routing apart from paging.

The arithmetic takes ten seconds and most candidates stall on it. The stall is the signal: it means error budgets are something you have read about rather than used.

Alert fatigue is a design failure, not a discipline failure. The fix is a rule about what may page at all, and burn-rate alerting is how you implement it.

Day-one telemetry is cheap and retrofitting it costs an incident. The answer that scores names the linking layer (trace IDs in logs, exemplars on metrics) that turns three tools into one story.

The roles question is really a delegation test: can you split commanding the incident from fixing it? Name the roles, the comms cadence, and the handoff discipline that keeps 3am coherent.

Spending the budget is arithmetic; enforcing consequences is politics plus automation. The answer that lands names trigger thresholds, the deploy gate that blocks releases, and the dated escape valve.

Facilitation is mechanics: timeline before theories, blame converted into system questions, actions with owners tracked to closure. Include the defensive engineer and the exec who wants a head.

Internal platforms hide a trap: the direct user is an engineer, but the real traffic belongs to their customers. Define SLIs from the consumer's vantage and turn the error budget into roadmap currency.

Monitoring dies silently: stuck alert queues, full TSDBs, collectors killed inside the same blast radius. Layer independent checks and a heartbeat so absence of signal pages louder than any red dashboard.

Tracing tells you which service and which span. It stops at the function boundary, and that is exactly where a CPU regression lives. Continuous profiling is the layer most teams do not have when they need it.

HTTP success can carry a rejected-span count. Replaying that batch can duplicate accepted telemetry, while enlarging the queue cannot repair invalid data.

The most-asked scenario question in DevOps interviews, and the one where candidates most often list commands instead of reasoning. Empty logs is the clue, not the obstacle.

A senior favourite, because the naive answer is to look at averages and the average is exactly what is hiding the problem. Tail latency has a short list of causes and they are all structural.

Tests whether you know that Kubernetes waits, and for how long. The default timings surprise most candidates, and they explain why a node failure hurts longer than people expect.

Both can be right, which is the whole point. Retransmissions mean a segment was not acknowledged, and the network is only one of the places that can happen.

Low utilisation with terrible tail latency is not a contradiction. It is the CFS quota doing exactly what it was configured to do. Most candidates have never read nr_throttled.

A two-second dependency should cost you latency, not three services. The mechanics are arithmetic: Little's law plus retry amplification. The fix is a budget, not a ban.

Repeated action items are a measurable defect with a mechanical cause. Candidates who answer 'culture' without a mechanism are describing the symptom back to you.

One disk metric explaining a whole cluster's weirdness is the point of this question. Candidates who restart etcd first turn a slow cluster into a down one.

Gradual drift produces no errors on the host that has it, only weirdness everywhere else. Sweep first, histogram second, repair third: the distribution shape names the culprit before you log into anything.

Slow leaks hide inside restart cycles and percentage thresholds. Plot weeks of absolute memory against deploys, prove it with profiler deltas, and alert on predicted time-to-limit instead.

Fork failures can reflect PID limits or memory allocation failure even when host RAM looks available. Find which ceiling broke, cap PIDs per cgroup so one team's leak costs one pod, and fix the unbounded spawn.

The pods are Pending, the nodes are Ready, and the GPUs have vanished from allocatable. Something changed on the node under a fleet that nobody considered part of the deploy surface.

Three terms that get used interchangeably and answer three different questions. The one that makes them operational is the admission rule at the end: no provenance, no deploy.

Choose by trust boundary, not fashion: mesh-issued mTLS inside a cluster, audience-bound tokens across teams and clouds, both together at scale. Static shared secrets are the thing being replaced.

The classic napkin-math question. The number in the prompt is the question, and candidates who draw the architecture before doing the arithmetic design something the physics forbids.

The Google-style NALSD round in miniature. The boxes are easy; what is being scored is whether the numbers fall out of Little's law before you draw them.

Cron is the most-deployed scheduler on earth and the worst-behaved at scale. The scored content is everything crontab cannot do: exactly-once firing, thundering herds, and what happens when the scheduler itself dies.

CAP is not a menu of two letters per system. The scored answer knows what the theorem claims (narrowly), what most engineers misquote, and what a split-brain costs when both halves accept writes.

Type-ahead is a latency budget wearing a search costume: keystrokes arrive every 200 milliseconds, so anything slower feels broken. Prefix indexes, freshness tiers, and typo tolerance are the scored parts.

Drawing an agent that polls a server is the easy third. The interview is about partial applies, fleet-versus-world state, and why rollback must be a pointer flip, never a reverse edit.

Reciting both acronyms is the median answer. Expert level chooses by connectivity assumptions and undo semantics, then defends CRDT metadata growth with numbers.

The question that separates people running GPU clusters from people running Kubernetes clusters that happen to have GPUs. The answer is that an integer count cannot express what a GPU workload needs.

Three sharing modes with three different isolation guarantees. MIG partitions the hardware, MPS shares the SMs, time-slicing just interleaves contexts, and picking wrong turns a latency SLO into a lottery.

The default scheduler places pods one at a time, which is exactly wrong for a job that needs all sixteen or none. Two half-placed jobs can hold each other's GPUs forever without either ever starting.

The driver, the container toolkit, the CUDA runtime inside the image and the framework build all have to agree. Upgrade the wrong one first and every pod on the node fails to start with an error that names none of them.

At sixteen GPUs and above, the network is the computer. Collectives run at the speed of the worst link in the group, and placement decides which link that is.

GPU capacity is not elastic the way CPU is. On-demand means whatever is left after the reservations, and in a tight region that is nothing, so the availability problem and the cost problem are the same problem.

The fastest-growing design round in infrastructure hiring. GPU economics break the assumptions CPU autoscaling is built on, and the answer has to start from that rather than from Kubernetes.

CPU memory intuition fails on inference servers. The scarce resource is KV-cache, requests arrive with unseen context lengths, and the fix is queue-depth scaling plus admission control, not bigger pods.

Code rollouts ask 'does it work'; model rollouts ask 'is it better', and the second question has no HTTP status code. Shadow traffic, offline eval gates and quality canaries replace the 500-rate dashboards you cannot use.

The utilisation number everyone quotes means at least one kernel was resident, not that the hardware was doing useful work. A GPU at 95 percent can be doing five percent of what it is capable of.

Code and data are the two things people pin, and they are rarely the two things that changed. The driver, the framework build, the kernel selection and the data ordering all vary underneath a pinned repository.

Failure stops being an event and becomes a rate. Once mean time between failures drops below the length of a run, the design question is not how to avoid interruption but how cheaply you can absorb one.

A release-automation design question with a specific trust boundary. The novel risk is prompt injection: untrusted text reaching a privileged agent, where the text is the instruction and your usual input validation does not apply.

Untrusted text reaching a privileged agent is the whole vulnerability. It is the pull_request_target mistake again, with a model in the middle that will follow instructions it finds in the data.

GPUs are the default for reasons that hold for large generative models and not for everything else. A small classifier on CPU can be cheaper, more available and easier to operate, and knowing where the crossover sits is worth more than knowing the newest card.

A model artifact is code that runs in your cluster with GPU access. Most of the supply-chain controls you already run for containers apply, and the file format adds one of its own.

An incident answer needs your actual decisions, the evidence behind them and the effect on users. A wrong turn belongs in the story only if it happened.

The most revealing behavioral question there is, because the interesting part is what you did after losing. Most candidates answer a different question, the one where they were right all along.

A question that punishes both extremes. A trivial mistake reads as evasion, a catastrophic one told carelessly reads as reckless, and the interviewer is really asking what you do in the twenty minutes after you realise.

Leadership rounds love this because it reveals whether you can separate ideas from egos in real time. The strong move is structural: change what is being decided, not who wins.

The defensiveness test. Interviewers score whether you can extract the one true behavioral signal from stinging delivery, change something observable, and close the loop with the person who said it.

Culture change with receipts. Interviewers score whether you built mechanisms, credited messengers, shipped postmortem actions, measured report latency, rather than reciting the blameless poster.

Arguing that reliability matters loses to a roadmap every time, because both sides agree it matters. What wins is converting it into the same currency features are argued in: a number, a cost, and a decision someone owns.

Two different resources produce the same errno, and the second one is invisible unless you know to look. One command distinguishes them.

A beginner question with a senior follow-up hiding in it. Once you know a hard link is a second name for an inode, log rotation and backup behaviour both stop being mysterious.

Everyone recites the four letters. The scoring answer walks each packet across the wire and knows the places the exchange silently dies.

Interviewers use this to sort memorised trivia from people who have chased a slow lookup. Three roles, one bit, and the caching rules most candidates repeat incorrectly.

A routine cloud operation with three layers that can quietly disagree. The strong answer is an ordered procedure you could run during business hours, without a reboot.

The answer is yes, and then the real question starts. Asymmetric routing and silent drops punish anyone who treats the routing tables as one flat list.

A growing backlog of sockets waiting for the application to close its side usually points at lifecycle handling. Here is the hunt from ss output to the pool timeout that caused it.

Most hand-written units boot in dev and die in production. The gap is Type selection, restart policy, and the drop-in workflow vendors expect you to respect.