Linux filesystem exhaustion: free bytes, inodes and deleted open files

TL;DR: Check the filesystem containing the failing path, then distinguish exhausted data blocks, exhausted inodes, quotas and space held by open files. Each consumes a different resource and needs different evidence.

A filename is only one reference

An inode records a filesystem object's metadata and references to its data. Directory entries associate names with inodes. Multiple hard links can name the same inode; an inode number is meaningful together with its filesystem identity. Linux documents these relationships in inode(7).

Filesystems differ in how they provision metadata. A filesystem with a limited inode supply can refuse new files while data blocks remain available. A build agent creating millions of tiny dependency-cache files is a realistic candidate. Increasing the disk's byte capacity does not automatically mean the existing filesystem gains the right metadata capacity; the filesystem type and resize procedure decide that.

Deleting a name also does not guarantee immediate space reclamation. When the last link is removed but a process still has the file open, the object remains available through that descriptor until the final open reference closes. See unlink(2). This explains why deleting an active log can make it disappear from a directory while filesystem usage barely changes.

Start at the failing path

Use df -h /var/lib/example for byte capacity and df -i /var/lib/example for inode counts, substituting the actual failing path. A host can have a mostly empty root volume while a separate application mount is full. Container writable layers, bind mounts and network filesystems add more reasons to identify the mount first.

du -x estimates usage reachable through the directory tree while staying on one filesystem. It does not count an unlinked file held only by a running process. Sparse files, permission errors and concurrent writes can create other differences between reports. Read both the command's output and its errors before concluding that space has vanished.

A bounded open-file experiment

This Python example creates a small temporary file and removes its directory entry while the handle remains open. It proves the reference-lifetime behavior without exhausting a filesystem or touching a service log.

import os
import tempfile
from pathlib import Path

with tempfile.TemporaryDirectory() as directory:
    path = Path(directory) / "worker.log"
    with path.open("w+b") as handle:
        handle.write(b"sample\n" * 32)
        handle.flush()
        path.unlink()
        assert not path.exists()
        assert os.fstat(handle.fileno()).st_nlink == 0
        handle.seek(0)
        assert handle.read().startswith(b"sample\n")
    assert not list(Path(directory).iterdir())

In an incident, use the affected process's descriptors or an approved lsof +L1 inspection to identify deleted files still open. Permissions may restrict visibility. Ask the service to reopen its log through its documented mechanism, or perform a controlled restart if that is the appropriate recovery path. Arbitrarily truncating a process's descriptors risks corrupting unrelated data.

Choose cleanup that addresses the cause

Consider an illustrative CI volume with 80 GB free but no available inodes. One million abandoned directories and small cache entries explain the failure better than a 20 GB archive. Removing the archive frees bytes but very few inodes. An age-based cleanup of identified, unused cache entries may reclaim the needed objects; deleting currently active workspaces can instead break builds.

Preserve enough evidence to connect growth to an owner and retention policy. A scheduled cleanup is incomplete if every job leaks temporary files faster than cleanup removes them. Monitor both byte and inode headroom where the filesystem exposes those limits, and budget for the largest expected burst.

Self-check: an operator deletes a 12 GB log, yet df remains unchanged and the service keeps logging. What evidence would justify restarting it? Confirm that the affected process holds the deleted inode and that this file accounts for the missing space. Then plan the service's recovery and log-reopen behavior. The df discrepancy alone is insufficient evidence to restart an unrelated process.