DevOpsInterviewPrep logo
Incident Response & Production Debugging / 02
easyNewRed HatIBMTCS

You are on a Linux box you have never seen and something is failing. Where are the logs?

Three places, and which one depends on how the thing was started. Knowing that a container's logs are not in /var/log, and that journald may not persist across reboots, saves the first twenty minutes.

Updated Sep 2026 · Grounded in researched DevOps, SRE and platform engineering interview loops, written to a senior-engineer editorial bar, and never padded to hit a word count.

TL;DR: journalctl -u <unit> for anything systemd started, /var/log/ for services that write their own files, and the container runtime for anything containerised. Check whether journald is persistent before you go looking for yesterday, because the default on some distributions keeps nothing across a reboot.

How to approach it

Answer by how the process was started, because that determines where its output went. Then give the flags that actually get used rather than a list of everything journalctl accepts. Mentioning the persistence question unprompted is the detail that signals you have lost logs to it before.

A strong answer

systemd units. Anything started by systemd has its stdout and stderr captured by journald.

journalctl -u nginx -n 100 --no-pager   # last 100 lines for one unit
journalctl -u nginx -f                  # follow
journalctl -u nginx --since "10 min ago"
journalctl -p err -b                    # errors only, this boot
journalctl -b -1                        # the previous boot
journalctl -k                           # kernel messages

-b for the current boot and -b -1 for the previous one are what you use after an unexplained reboot. -p err cuts a wall of informational noise to the lines someone should read.

The thing to check early: journald's storage. If /var/log/journal/ does not exist, journald is running in volatile mode and keeps logs in memory only, so a reboot loses everything. That is the default on some minimal images and it is the reason a machine that just rebooted has nothing to say about why.

ls -d /var/log/journal 2>/dev/null || echo "volatile: logs do not survive a reboot"

Making it persistent is creating that directory and restarting journald, and it is worth doing before you need it.

Files under /var/log. Applications that manage their own logging write here, and the layout depends on the distribution. /var/log/syslog or /var/log/messages for general system messages, /var/log/auth.log or /var/log/secure for authentication, and per-application directories like /var/log/nginx/ and /var/log/postgresql/. When you do not know what a service writes, ask the package:

ls -lt /var/log | head -20        # what changed most recently
lsof -p <pid> | grep -i log       # what this process actually has open

lsof on the process is the reliable answer when documentation and convention disagree, which they often do for anything installed from a vendor tarball.

Containers. Logs are not in /var/log inside the container in any well-behaved setup, because the convention is to write to stdout and let the runtime collect it.

docker logs -f --tail=100 <container>
kubectl logs -n prod deploy/checkout --tail=100
kubectl logs -n prod checkout-7d9f --previous   # the container that just died
kubectl logs -n prod deploy/checkout -c sidecar # a specific container in the pod

--previous is the one people forget, and it is the only way to see why a crash-looping container failed, because the current attempt may have written nothing yet.

On the host, the runtime writes those streams to files under /var/log/pods/ and /var/log/containers/ on a Kubernetes node, which is useful when the API server is unreachable and kubectl is not an option.

Two more habits. Check dmesg -T early for anything hardware, kernel or OOM related, since a process killed for exceeding its cgroup memory limit leaves nothing in its own logs and a clear line in the kernel ring buffer. And check the clock and timezone before comparing timestamps across machines, because an hour's offset sends people down the wrong path faster than almost anything else.

What interviewers probe next

"The disk filled with logs. What do you do?" Rotate rather than delete, because deleting a file a process still holds open frees nothing. logrotate with copytruncate or a postrotate signal so the daemon reopens its file.

"How do you follow one request across services?" You do not, from files. That needs a trace ID in every log line and a central store, which is the argument for shipping logs somewhere rather than reading them per host.

"What is the difference between journald and rsyslog?" journald is systemd's structured binary store queried with journalctl. rsyslog is the traditional daemon writing plain text under /var/log, and many systems run both, with journald forwarding to rsyslog.

Common mistakes

Looking in /var/log inside a container, where a correctly configured application writes nothing.

Reading kubectl logs without --previous on a crash loop and concluding there is no output.

Assuming journald kept yesterday, when volatile storage discarded it at the reboot you are investigating.

Ignoring dmesg, which is where the OOM killer records the process it terminated.

That one was free, and so are 18 answers per topic without an account. Signing in doubles that to 28, keeps your bookmarks, and tracks which topics you keep getting wrong.one Google click · no card · nothing to cancel
HOW DID IT GO?
0
UP NEXT ON YOUR JOURNEY
DISCUSSION · 0

Nothing here yet. Say how you would answer it.