File descriptor exhaustion: distinguish process limits, leaks and connection demand

TL;DR: Identify the process and the failing syscall, then compare its open descriptors with its effective limit. Raise a justified capacity limit only after distinguishing legitimate concurrency from handles that remain open after their work ends.

A descriptor names an open resource

A file descriptor is a small integer in a process's descriptor table. Regular files, sockets and pipes all consume entries. Opening a socket does not make it exempt from file limits. Duplicating a descriptor creates another table entry even when both refer to the same underlying open file description; inheritance across process creation makes system-wide counting more subtle than adding every process's list.

Linux RLIMIT_NOFILE sets the process's descriptor-number boundary, one greater than the maximum descriptor number it may open. The soft value is enforced; the hard value bounds ordinary changes to the soft value. See getrlimit. A shell's ulimit describes that shell and future children, not an unrelated service already started by systemd.

An EMFILE failure commonly indicates the calling process's limit. ENFILE indicates exhaustion of the system-wide open-file table. The open syscall reference documents both. Record the actual error instead of translating every failure into “too many connections.”

Count what the affected process holds

For a Linux process you own, inspect /proc/<pid>/limits and the entries in /proc/<pid>/fd. Confirm the PID's identity and start time so a recycled PID does not mix two services. Socket descriptors appear as links to socket inodes; correlate them with the process's connection inventory when permissions allow. The proc_pid_fd reference explains this view.

A terminated TCP connection can leave kernel state such as TIME_WAIT without retaining an application descriptor. Conversely, a socket in CLOSE_WAIT can indicate that the peer closed and the application has not completed its own close. Neither state count substitutes for the process's actual descriptor table.

Budget a proxy before changing its limit

Assume a hypothetical proxy has a soft limit of 4,096. Its listeners, logs and internal connections consume 120 descriptors. Each active downstream connection also opens one upstream connection, so this simplified design uses two descriptors per active request connection. Reserve 400 descriptors for control operations and bursts.

The budget is floor((4,096 - 120 - 400) / 2) = 1,788 active pairs. At 1,900 pairs, the process holds about 3,920 descriptors before temporary work. A burst can exhaust the remaining space even when CPU and memory look comfortable.

Real HTTP/2 multiplexing, upstream pooling and idle connections change this relationship. Measure the implementation's connection model, then size its application limits consistently. Raising nofile alone can transfer the bottleneck to upstream pools, kernel memory or a downstream database.

Reproduce EMFILE in a disposable process

Run this standard-library script on Linux as a separate process. It lowers only its own soft limit and opens at most 80 handles to /dev/null, releasing them in finally. Do not run it inside a long-lived application or notebook kernel.

import errno
import os
import resource

soft, hard = resource.getrlimit(resource.RLIMIT_NOFILE)
limit = 64 if soft == resource.RLIM_INFINITY else min(soft, 64)
resource.setrlimit(resource.RLIMIT_NOFILE, (limit, hard))
handles = []
try:
    for _ in range(80):
        handles.append(os.open("/dev/null", os.O_RDONLY))
except OSError as exc:
    assert exc.errno == errno.EMFILE, exc
    print("EMFILE after", len(handles), "new handles")
finally:
    for handle in handles:
        os.close(handle)

The number printed depends on already-open descriptors. Expect fewer than the soft limit, not exactly 64 new handles. This tests a process-local boundary; it deliberately does not exhaust the host's file table.

An open descriptor can keep a deleted file alive. Filesystem and inode exhaustion explains why removing a log name may leave disk usage unchanged.

Self-check: increasing the service limit stops errors, but descriptor count still climbs by 100 after every completed batch. Is the fix complete? No. The trend remains unbounded. Find which resources survive each batch and verify a stable post-work baseline after repairing cleanup. Explain the higher limit as temporary headroom in an interview, with the observed retention as the unresolved cause.