DevOpsInterviewPrep logo
Linux, Networking & Scripting / 06
mediumNewAmazonRed HatTCS

Writes are failing with no space left on device, but df shows 40% free. What is going on?

Two different resources produce the same errno, and the second one is invisible unless you know to look. One command distinguishes them.

Updated Sep 2026 · Grounded in researched DevOps, SRE and platform engineering interview loops, written to a senior-engineer editorial bar, and never padded to hit a word count.

TL;DR: You are out of inodes, not blocks. df -h reports data blocks; df -i reports inodes, and a filesystem with free space and zero free inodes cannot create another file. It is caused by very large numbers of very small files, and the expansion options depend on the filesystem.

How to approach it

Name the second resource immediately and give the command that proves it. Then check filesystem growth support before deciding whether adding storage helps, which is the part that makes it interesting.

A strong answer

An inode holds a file's metadata: permissions, owner, timestamps, size and the pointers to its data blocks. The filename lives in the directory entry, not the inode. On ext4 the inode table is allocated at format time from a ratio (one inode per 16KB of space by default), so density is set at creation; growing ext4 adds block groups and their inodes. Create more files than that count, however small, and creation fails with ENOSPC even though blocks remain.

df -h /var      # 40% free  <- blocks
df -i /var      # 100% used <- inodes, this is your answer

Then find them. Counting files per directory is the practical move:

du --inodes -x /var | sort -rn | head
# or, on older systems:
find /var -xdev -printf '%h\n' | sort | uniq -c | sort -rn | head

The usual sources: a session or cache directory nobody prunes, a mail queue, millions of tiny log files from a process that opens a new one per event, unpacked container layers, or a directory of temporary files whose cleanup job has been failing silently for months.

The fix is deleting files, and that is the awkward part, because deleting millions of small files is slow and a single rm over a huge directory can take hours and hammer the disk. A recursive removal still unlinks its children and can generate heavy metadata I/O. Confirm the data is disposable, remove in controlled batches and monitor storage latency. For prevention: rotate and compress, cap the count rather than only the size, and consider whether the workload should be writing that many files at all.

Two structural notes worth having. XFS allocates inodes dynamically, so it does not have this failure mode in the same way, which is a genuine argument for it on workloads with unpredictable file counts. Growing ext4 adds inodes at its existing density. If that cannot supply enough capacity, a planned migration to a filesystem created with a lower bytes-per-inode ratio is another option, requiring verified backup and restore; never run mkfs on the live filesystem.

What interviewers probe next

"How would you monitor for it?" Alert on inode usage alongside block usage. Most default dashboards only show blocks, which is precisely why this comes as a surprise.

"What is a hard link, in this context?" Another directory entry pointing at the same inode. Many names, one inode, so hard links do not consume additional inodes and the data survives until the last link is removed.

"Same symptom, different cause?" A deleted-but-open file holds blocks, which produces the reverse: du shows space free and df does not.

Common mistakes

Growing only the block device without checking whether the filesystem was expanded and gained usable inodes.

Never running df -i, which leaves you unable to explain a plainly contradictory df -h.

Deleting files one by one across millions of entries and causing a second incident with the I/O load.

References

resize2fs growth.

That one was free, and so are 10 answers per topic without an account. Signing in doubles that to 20, keeps your bookmarks, and tracks which topics you keep getting wrong.one Google click · no card · nothing to cancel
HOW DID IT GO?
0
UP NEXT ON YOUR JOURNEY
DISCUSSION · 0

Nothing here yet. Say how you would answer it.