Microsoft SRE / Production Engineer interview questions

The answer is yes, and then the real question starts. Asymmetric routing and silent drops punish anyone who treats the routing tables as one flat list.

The signature is unmistakable once seen: handshakes complete, then the connection freezes at the same byte count every time. Here is the probe sequence and the fixes, ranked.

Table full is the symptom everyone knows. The interview answer lives one layer down: per-state timers, the five-day ESTABLISHED default, and what conntrack -S says about hash pressure.

Asked in almost every cloud screen, and most candidates recite the table without ever naming the failure it causes. The ephemeral port range is the whole answer.

Most candidates can sell serverless. The question asks you to argue against it, which needs a real understanding of the execution model rather than the marketing.

The junior-filter question that still separates candidates who understand the model from those who memorised a definition. The answer is three mechanical differences and one economic one.

Four options with genuinely different costs, and the requirement is usually narrower than it first sounds. Establish whether it is about the network path, the data residency, or the provider's retention terms, because those need different answers.

The private endpoint can be healthy while an office resolver still returns the public address. Trace the lookup from the failing client before changing access rules.

Moving a Kubernetes service account changes its token subject. Separate pod mutation, federation trust and Key Vault authorization before adding permissions.

General-purpose IaC earns its complexity where application engineers own infrastructure and benefit from typed components and unit tests. It disappoints when the pitch was freedom: the engine still builds a graph, still keeps state, and still forbids most of the clever runtime logic people expect to write.

Distinguish a gated Terraform run workflow from a cloud controller that reconciles continuously. Both can start from Git, but they expose different approval, drift and credential boundaries.

The pitch is continuous reconciliation instead of a plan someone runs. The cost is that your cloud estate now depends on a cluster, and that drift correction without a plan step is a very different risk posture.

It sounds like trivia and screens for something real: whether you understand that a pod is a shared namespace boundary with an owner, not just a group of containers.

SSH plus yum is how outages start. The senior answer replaces nodes from versioned images on a cadence, orchestrates with surge capacity and PDBs, and can explain the kernel-skew problem that makes fleet homogeneity a feature.

Kubelet resource metrics to metrics-server to aggregated API to HPA controller: every hop adds lag and averaging. Walking the path and naming where it distorts demand separates operators from dashboard readers.

Heartbeats, conditions and taints are the only vocabulary the control plane understands. NPD translates journald and kernel ring messages into that vocabulary; custom plugins carry your hardware's dialect.

Kubernetes does not see GPUs, it sees an extended resource that something has to advertise. When that advertisement is missing or wrong, the scheduler is behaving correctly and the node is lying.

A Job with a sidecar was broken by design for years: the workload finishes, the proxy does not, and the pod never terminates. Native sidecars fix it by making the container an init container that keeps running.

Changing pod resources historically required recreation, which made vertical autoscaling disruptive for workloads with expensive recovery. In-place resize changes that, with limits worth knowing.

Ingress solved routing by convention and annotation, which meant every controller invented its own. Gateway API's real contribution is not features, it is splitting one object into three so that platform and application teams stop editing the same file.

Clusters multiply for good reasons: regions, isolation, tenancy, blast radius. The problem is not creating them, it is that cluster seven diverges from cluster three and nobody knows until an upgrade behaves differently.

Route attachment and cross-namespace backend access have different owners. Use route status to find the failing boundary before broadening any permission.

The fetch and pull half is a warm-up. The rebase half is where candidates either show they understand that rebasing rewrites commits or reveal that they have only ever followed instructions.

Most candidates narrate the YAML. The scoring starts underneath: who schedules what, where artifacts live between jobs, and why disposable runners rewrite how you think about caching and trust.

Deployment and release are two decisions, not one. The flag patterns that buy you instant reversibility, and the retirement discipline most teams skip.

Quarantine is a holding cell with an expiry date, not a landfill. Per-test flake rates, entry evidence, an enforced expiry review, and the cap that turns fixing into priority.

Green PRs can still break main because they were validated against a stale tip. Merge queues revalidate the exact combination that lands, trading latency for certainty.

Reciting metric names scores low. Naming the definition decisions that corrupt each one, and the gaming that starts when targets appear, scores high.

Build graphs identify affected targets only when their dependencies are complete. The ladder from build-everything to content-addressed caching, hermeticity audits, and where undeclared inputs rot correctness.

One cadence for every team is the naive answer. Cadence follows constraints: store review for mobile, compatibility windows for APIs, contracts and embedded realities for trains.

The code did not change, the data did. That inverts what triggers the pipeline, what the tests assert, and what a rollback means.

A token naming the right repository does not prove that the approved deployment workflow ran. Bind the caller, deployment context and called workflow into the cloud trust contract.

MTTR is three clocks hiding in one number. Profile detection, diagnosis and fix separately from incident timestamps; attack whichever clock dominates, which is almost never rollback speed.

An SLA is a price list for failure signed before the failures. Bring the measured distribution, negotiate methodology and exclusions harder than the headline number, and phase targets with dates.

Agent per node, gateway per cluster, and one decision that shapes everything else: tail sampling needs every span of a trace in one place, which is why the topology has a tier that looks redundant until you need it.

HTTP success can carry a rejected-span count. Replaying that batch can duplicate accepted telemetry, while enlarging the queue cannot repair invalid data.

Emergency mode after a storage change is usually one line in fstab. What interviewers score is your order of operations and whether your recovery preserves data-mount dependencies.

Load balancing makes a per-host fault look like a flaky library. The exact rejection string tells you which way the offending clock leans before you log into anything.

Gradual drift produces no errors on the host that has it, only weirdness everywhere else. Sweep first, histogram second, repair third: the distribution shape names the culprit before you log into anything.

Bandwidth graphs look healthy and the partner swears they are fine. The wall is ports, not bits: the available source-port budget depends on the gateway and its address allocation.

Purges act on exact cache keys, and the edge is full of variants the purge never named. Verify per variant at real POPs, then fix the key model instead of chasing URLs forever.

Stateful firewalls keep established flows alive after their rule vanishes, so cleanup day passes every check and lag appears days later. Flow logs name the hour; inventories prevent the repeat.

Wall-clock schedules meet daylight-saving transitions: spring forward deletes an hour of cron targets, fall back repeats one. Schedule in UTC, add idempotency keys, rehearse in staging.

Queue time, prefill and decode fail for different reasons and the aggregate hides which. Split the metric first and the candidate list drops from twenty to three.

A rebuilt index and a wrong answer means the staleness is somewhere else: a cache, a routing pointer, a chunk that survived deletion, or a document that never reached the pipeline at all.

The pods are Pending, the nodes are Ready, and the GPUs have vanished from allocatable. Something changed on the node under a fleet that nobody considered part of the deploy surface.

Threat modelling dies as a document ritual and lives as a design-stage habit. The answer that scores: tier reviews by blast radius, short sessions on RFCs, findings filed where engineers already work.

Push approval is a weak factor and fatigue is its predictable failure mode. Stop the campaign with number matching and rate limits; end the attack class with passkeys, device gating and a hardened helpdesk.

Labels without consequences die quietly. Attach classification to enforced defaults in Terraform modules, propagate tags to every copy of the data, and give engineers twenty worked examples instead of a taxonomy lecture.

An agent is a workload that acts on behalf of a person, holds credentials for several systems, and decides what to call based on text it read. Every one of those properties breaks an assumption in your existing access model.

Model output is attacker-influenced data by construction, because the model reads attacker-influenced input. Every rule you already apply to user input applies here, and teams forget because the text came from a system they built.

The most rehearsed design question, which is exactly why interviewers score the parts blog posts skip: redirect semantics, key generation under failure, and analytics kept off the hot path.

The difficult decisions are preserving offline intent and enforcing rules that span devices. Interviewers want conflict classes named and handled deliberately, not a CRDT framework proposed on instinct.

Reciting both acronyms is the median answer. Expert level chooses by connectivity assumptions and undo semantics, then defends CRDT metadata growth with numbers.

Teams need comparable evaluations before they approve a model change. Design versioned test data, recorded runs and release thresholds that a pipeline can enforce.

Data transfer appears on nobody's architecture diagram and reliably appears on the invoice. The strong answer enumerates the taxed flows, ranks the levers, and knows which tax is worth paying.

An upfront purchase, a refund and two currencies can produce several valid totals. Work out which total finance needs, then prove every adjustment from the source rows.

Three sharing modes with three different isolation guarantees. MIG partitions the hardware, MPS shares the SMs, time-slicing just interleaves contexts, and picking wrong turns a latency SLO into a lottery.

GPUs fail gradually before they fail loudly. Xid errors, ECC retirement, thermal throttling and a degraded NVLink all produce a node that is up, schedulable, and slowing every gang it joins.

The driver, the container toolkit, the CUDA runtime inside the image and the framework build all have to agree. Upgrade the wrong one first and every pod on the node fails to start with an error that names none of them.

Hard quotas strand idle GPUs and soft quotas mean the loudest team owns the cluster. Borrowing with reclaim is the shape that works, and the interesting decisions are all about what happens when the lender comes back.

At sixteen GPUs and above, the network is the computer. Collectives run at the speed of the worst link in the group, and placement decides which link that is.

GPU capacity is not elastic the way CPU is. On-demand means whatever is left after the reservations, and in a tight region that is nothing, so the availability problem and the cost problem are the same problem.

A mixed fleet is the normal state, not a transitional one, because capacity arrives when it arrives. The failure is a scheduler that treats two very different cards as the same countable unit.

Both configurations fit the weights. Compare cache capacity, collective overhead and replica placement under the actual traffic shape; neither layout wins every workload.

Separate scheduling delay from image pull, artifact transfer, GPU loading and warm-up. The autoscaler needs the full request-to-ready time.

The infrastructure win is real and easy to measure. The quality regression is real, harder to measure, and will not show up in your latency dashboards at all.

It reads as a configuration change and it is a capacity change. Cache memory per request grows with the limit, prefill cost grows with what people actually send, and concurrency falls in proportion.

Prefill and decode are two different workloads sharing one GPU. Their resource demands differ, and an unchunked long prefill can delay active decodes. Profile the actual model and batch shape before choosing a scheduling change.

Compatible LoRA adapters let tenants share resident base weights. Check adapter size, KV-cache demand and traffic before sizing the replica pool.

It trades compute for latency, so it wins on an underloaded fleet and can lose on a saturated one. The acceptance rate decides everything, and the acceptance rate depends on traffic you do not control.

Scale to zero is correct for some workloads and an outage generator for others, and the deciding number is what the first request after idle experiences. Cold start turns an availability promise into a queue.

A request that streams for ninety seconds is not slow, it is long. Total latency mixes the part users wait for with the part they read, and an SLO built on it will page you for success.

One request touches an embedding call, a vector search, a rerank, a prompt build and a generation, and a single duration for the whole thing tells you nothing about any of them.

Prompts contain whatever the user typed, which in a support product is names, account numbers and medical detail. Full-text logging is a data-protection decision, not a debugging preference.

Latency is flat, errors are zero, and the answers are worse. Quality regression is invisible to every metric a platform team normally owns, which is why it needs its own pipeline.

The training part is a job. Everything around it is the pipeline: data snapshots that are immutable, an evaluation gate with a threshold set in advance, and adapters versioned so a bad one is a routing change away from gone.

Checkpoint too rarely and a failure costs hours of GPU time. Checkpoint too often and you pay for stalled GPUs writing terabytes to storage. There is an arithmetic answer and most teams never do it.

Failure stops being an event and becomes a rate. Once mean time between failures drops below the length of a run, the design question is not how to avoid interruption but how cheaply you can absorb one.

A model is an artifact with a lineage, an evaluation record and an owner, or it is a file somebody copied to a bucket. The registry exists so that the second one cannot reach production.

The database part is easy. The operational problems are reindexing when the embedding model changes, recall degrading silently as filters get stricter, and memory growing faster than anyone forecast.

A release-automation design question with a specific trust boundary. The novel risk is prompt injection: untrusted text reaching a privileged agent, where the text is the instruction and your usual input validation does not apply.

Untrusted text reaching a privileged agent is the whole vulnerability. It is the pull_request_target mistake again, with a model in the middle that will follow instructions it finds in the data.

The valuable part is the first ten minutes of context gathering, which is tedious and mechanical. The dangerous part is anything that changes production while a human is still working out what is wrong.

Generated infrastructure code is plausible, fluent and confidently wrong in specific ways. The volume goes up, the reviewer attention per line goes down, and the controls that used to be optional stop being optional.

Every team standing up its own tool server produces a permission surface nobody can enumerate. The gateway exists so that connecting an agent to a system is a reviewed act rather than a config file in somebody's repository.

GPU cost attribution needs allocation records as well as utilization. Separate exclusive reservations from borrowable quota, then reconcile team charges and idle capacity to the actual bill.

A notebook holds a GPU while somebody thinks, which is most of the day. The utilisation on an interactive fleet is the worst in the building, and the fix is a queue people do not hate.

GPUs are the default for reasons that hold for large generative models and not for everything else. A small classifier on CPU can be cheaper, more available and easier to operate, and knowing where the crossover sits is worth more than knowing the newest card.

A model artifact is code that runs in your cluster with GPU access. Most of the supply-chain controls you already run for containers apply, and the file format adds one of its own.

Shared batching puts two tenants' data in one process on one device. That is defensible, and it is not what most customers assume they bought, so the answer has to be specific about which boundary exists where.

A mandatory gateway can centralize limits and filtering, while applications authorize business actions. Neither a prompt nor a PII classifier guarantees that sensitive data stays private.

Most of the obligation lands on the product, but three parts land on infrastructure: an inventory of what is deployed, evidence of what it did, and the ability to turn it off. Those are platform capabilities and they take months to retrofit.

Leadership rounds love this because it reveals whether you can separate ideas from egos in real time. The strong move is structural: change what is being decided, not who wins.

New leads fail by hoarding work or dumping it. Interviewers want your decision rules for handing things over, what you deliberately keep, and how you avoid becoming the bottleneck.

The defensiveness test. Interviewers score whether you can extract the one true behavioral signal from stinging delivery, change something observable, and close the loop with the person who said it.

Use a real example if you have shipped an AI feature. If you have not, say so and offer a clearly hypothetical response about evaluation, containment and customer communication.

Turns the standard autoscaling definition into the problem it actually causes. Almost every candidate can define an HPA; far fewer can say why theirs did not save them.

The question that separates console users from people who have debugged a permissions failure across accounts. Evaluation order is deterministic and knowing it makes AccessDenied readable.

A design question with a clear answer at this scale, and the reasoning is what scores. The three options solve different problems and peering and Transit Gateway connect networks while PrivateLink exposes services.

Trace the route and return path before changing rules. First establish whether the failing connection uses IPv4, IPv6 or a private service endpoint.

S3 access failures are rarely about the bucket policy alone. The caller, object ownership and network path determine which authorization checks apply. Start with the failed request and trace changes across those boundaries.

Database redundancy still needs reconnecting clients. Distinguish the deployment type, then measure how promotion and DNS changes affect application requests.

Spreading instances across availability zones is not the same as surviving an AZ failure. The candidates who know this can name the four coupling points that quietly re-unify your blast radius.

The services-and-enterprise interview staple. There is a defensible answer and it is not 'always re-platform': the decision is a function of change tolerance, deadline reality and what the workload is worth.