DevOpsInterviewPrep logo
Incident Response & Production Debugging / 07
mediumNewGitLabAtlassianMeta

The pipeline says the deploy succeeded, but users still see the old version. Find it.

A deceptively deep question, because there are at least six places a version can get stuck and they span the pipeline, the cluster and the edge. Work the path in order.

Updated Sep 2026 · Grounded in researched DevOps, SRE and platform engineering interview loops, written to a senior-engineer editorial bar, and never padded to hit a word count.

TL;DR: Something between your build and the user is serving a cached or unchanged artifact. Walk the path in order and check identity at each hop: did the image build, did the tag move, did the pods actually restart, are the old pods still receiving traffic, and is a CDN or browser cache in front. The most common single cause is a mutable tag with imagePullPolicy: IfNotPresent.

How to approach it

Do not guess a layer. Say you will walk the path from build to browser and check a version identity at each hop, then name the hops. That structure is the answer; the specific bug varies.

A strong answer

Hop one, the artifact. Did the build actually produce a new image, and does the digest differ from the previous one? A pipeline that succeeded because a cached layer satisfied everything can produce an identical image. Compare digests, not tags.

Hop two, the tag. If you deploy myapp:latest or myapp:v2 and push a new image to the same tag, the manifest is unchanged from Kubernetes' point of view. With imagePullPolicy: IfNotPresent, a node that already has that tag cached never pulls again, so you get old code on old nodes and new code on new ones, which presents as an intermittent old version. This is the most common cause by a distance, and the fix is immutable tags: deploy by digest or by a unique build identifier, never a moving tag.

Hop three, did the rollout happen? A Deployment only rolls when the pod template changes. Changing a mounted ConfigMap does not restart the pod. Volume files normally update eventually and the application must reload them; subPath mounts do not receive those updates, and environment variables require a restart. Check kubectl rollout status and the ReplicaSet generation, and confirm pod ages are younger than the deploy.

Hop four, are old pods still serving? A partially completed rollout leaves both versions receiving traffic, so a user hitting the old one reports the old version while a colleague sees the new. Check whether the old ReplicaSet still has replicas.

Hop five, GitOps drift. If ArgoCD or Flux owns the manifest and your pipeline patched the cluster directly, the controller reverts it on the next reconcile. The pipeline reports success, the cluster agrees for ninety seconds, and then it does not. kubectl get deployment -o yaml twice, a minute apart, catches it.

Hop six, the edge. A CDN or reverse proxy caching HTML, or a service worker holding a stale bundle in the browser. The tell is that curl against the origin returns the new version while the browser does not.

The habit that prevents most of this: expose a build identifier at a health endpoint, and make the deploy verification step assert it. Then "deploy succeeded" means "the new version is answering", not "the apply returned zero".

What interviewers probe next

"Why is latest dangerous beyond this?" You cannot roll back to a known state, because the tag no longer points where it did. It also makes the running version unknowable from the manifest.

"How would you force a restart on a config change?" Roll a checksum of the ConfigMap into a pod annotation, so a config change changes the pod template and triggers a real rollout.

"What proves it at the end?" A version endpoint checked by the pipeline against the commit it deployed.

Common mistakes

Restarting the deployment first. It fixes the symptom on the spot and destroys the evidence of which layer was stale.

Trusting the tag. Digests are the identity; tags are a label someone can move.

Forgetting the GitOps reconcile loop, which produces the uniquely confusing version where it works and then un-works.

References

Kubernetes ConfigMaps.

That one was free, and so are 10 answers per topic without an account. Signing in doubles that to 20, keeps your bookmarks, and tracks which topics you keep getting wrong.one Google click · no card · nothing to cancel
HOW DID IT GO?
0
UP NEXT ON YOUR JOURNEY
DISCUSSION · 0

Nothing here yet. Say how you would answer it.