TL;DR: Stream it. Read line by line and keep only aggregates in memory, never the file. Then show the things that separate a script from a one-liner: a strict shell preamble, no unquoted variables, handling the empty case, and being safe to re-run. The memory limit is there specifically to catch anyone who reaches for a function that reads the whole file.
How to approach it
Say the word streaming immediately, then write something small and correct. The interviewer is not looking for cleverness; they are looking for the habits that stop a script destroying something at 3am.
A strong answer
The constraint rules out anything that materialises the file: file.read(), readlines(), $(cat file), sorting the whole thing in memory. What is allowed is a single pass holding bounded state.
For extracting 5xx responses and counting by client:
#!/usr/bin/env bash
set -euo pipefail
log="${1:?usage: $0 <logfile>}"
[[ -r "$log" ]] || { echo "cannot read $log" >&2; exit 1; }
export LC_ALL=C
awk '$9 ~ /^5[0-9][0-9]$/ { print $1 }' "$log" \
| sort -S 64M \
| uniq -c \
| sort -S 64M -k1,1nr -k2,2 \
| sed -n '1,20p'
This example assumes a known whitespace-delimited access-log schema with status in field nine. awk emits keys without retaining a map; GNU sort spills to disk with a 64MiB buffer target, and uniq counts adjacent keys. Budget temporary disk space and unusually long records too. sed consumes the full result, avoiding head-induced SIGPIPE under pipefail. A top-N heap alone cannot recover exact counts for unbounded distinct keys; approximate heavy-hitter algorithms must disclose their error bounds.
For a known bounded client population, this Python counter is simpler. Its memory grows with distinct keys; it does not satisfy an arbitrary-cardinality 512MiB guarantee:
import sys
from collections import Counter
counts = Counter()
with open(sys.argv[1]) as f: # iterating a file object streams it
for line in f:
parts = line.split()
if len(parts) > 8 and len(parts[8]) == 3 and parts[8][0] == "5" and parts[8][1:].isascii() and parts[8][1:].isdigit():
counts[parts[0]] += 1
for ip, n in counts.most_common(20):
print(n, ip)
Now the habits being scored, which matter more than the algorithm.
set -euo pipefail at the top: exit on error, treat unset variables as errors, and make a pipeline fail if any stage fails rather than only the last. Without pipefail, grep x file | sort succeeds when grep fails, which is how scripts silently produce empty results.
Quote every variable. An unquoted variable allows word splitting and glob expansion. An empty rm -rf $dir alone removes nothing, but a construction such as rm -rf "$dir"/* with an empty directory variable can target the root. Validate an allowed nonempty path before a destructive command; ${dir:?} rejects an unset or empty value.
Handle the empty and malformed cases. A log with zero matches should exit cleanly, not divide by zero or print a header with nothing under it. Real log files contain truncated final lines.
Make it safe to re-run. If it writes output, write to a temporary file and move it into place atomically, so an interrupted run does not leave a half-written result that the next stage consumes.
What interviewers probe next
"How would you parallelise it?" Split by byte offset on line boundaries and merge partial counters, or xargs -P over rotated files. Say the honest thing too: a single pass is usually I/O bound, so parallelism helps only when parsing is the bottleneck.
"What if it must run on a live file being written?" Read what exists and record the offset, or work on rotated files. Reading a file under active rotation gives you a truncated result and no error.
"Bash or Python?" Bash for pipelines of existing tools, Python once there is real parsing, data structures or error handling. Knowing where the line is is part of the answer.
Common mistakes
Reading the file into memory, which the constraint exists to catch.
Omitting the strict preamble, which is the fastest signal that someone writes scripts rather than tools.
Unquoted variables, which is the most common real bug in production shell and is visible in ten seconds of reading.
References