Linux processes and system calls: from application code to kernel evidence
Connect Linux processes, threads and system calls to production diagnosis. Interpret file descriptors, blocking calls, errno and tracing limits with a small local exercise.
TL;DR: A process runs application code in user space and requests kernel-managed operations through system calls, commonly via library wrappers. Diagnose a slow process by determining whether it is computing, waiting or repeatedly failing at that boundary before changing resource limits.
The kernel owns shared resources
Opening a file, accepting a network connection and waiting for a child involve kernel-managed resources. A library call is not always a system call: it may work entirely in user space, buffer data, use a cached result or invoke several system calls. The Linux system-call introduction explains the wrapper boundary and error reporting.
A process has an address space and resource context. Threads within it share many resources while having their own execution state. The kernel schedules runnable threads; a thread blocked waiting for I/O is different from a runnable thread waiting for CPU time. Both can contribute to application latency, but the corrective action differs.
The error and wait paths are separate possibilities. A nonblocking operation may return immediately with a condition the application must handle, while a blocking call may sleep until progress becomes possible.
File descriptors identify open handles
A file descriptor is a small integer in a process's descriptor table. It can refer to a file, socket, pipe or other supported object. The same integer in two unrelated processes need not refer to the same object. File-descriptor limits therefore need to be interpreted in the context of the affected process and service manager configuration.
If a server leaks accepted sockets, it can eventually fail to open new descriptors even though CPU and memory dashboards look ordinary. Raising the limit may buy time, but identify the lifecycle defect. Check whether connections close after timeout and whether cancellation paths release resources.
| Evidence | Possible interpretation | What would distinguish causes |
|---|---|---|
| High user CPU | Application or runtime computation | Profile the hot code path |
| High system CPU | Heavy kernel work or syscall rate | Inspect workload and kernel-side activity |
Repeated EMFILE | Process descriptor limit reached | Count handles and inspect connection/file lifecycle |
Repeated ENOENT | Path absent in the process's view | Check working directory, mounts and expected files |
| Long blocked reads | Waiting for input | Identify the descriptor and upstream producer |
Frequent EAGAIN on nonblocking I/O | Operation cannot proceed immediately | Check whether the event loop waits correctly or spins |
An error name describes the observed condition, not the full cause. A missing file inside a container can result from a mount or namespace difference even when the host path exists.
A small trace you can run locally
On a Linux machine with strace installed and tracing permitted, this command traces a new cat process reading a standard kernel information file. It does not attach to a production process.
strace -e trace=openat,read,write,close -o /tmp/kernel-read.trace \
cat /proc/version
Inspect /tmp/kernel-read.trace. You should see library/runtime setup as well as the target file operation, so the trace will contain more than “open one file, read one line.” Exact calls and descriptor numbers depend on the distribution and runtime. The strace manual documents syscall filtering and output behavior.
Do not paste an unfiltered production trace into a ticket. System-call arguments and buffers can include credentials, customer data and paths. Tracing also adds overhead and can alter timing. Choose a narrow scope, a bounded capture and appropriate access when investigating a live service.
Waiting is not automatically a CPU shortage
A worker blocked on a database socket may show low CPU usage and high request latency. Adding CPU does not shorten the database's response time. A worker repeatedly retrying a nonblocking operation without waiting can instead consume a core while making little progress.
Correlate thread state, syscall behavior, application traces and dependency metrics. One tool rarely establishes the complete chain. A syscall trace showing a long wait on a socket tells you where this process waited; it does not identify whether the remote database was CPU-bound, locked or unreachable through the network.
Container boundaries complicate observation. PID and mount namespaces change what a process can see, while cgroups govern resource accounting and limits. Use namespaces and cgroups to separate those mechanisms. A container shares the host kernel even when its userspace looks like another Linux installation.
Process creation and replacement
The familiar fork-and-exec model separates creating a process from replacing its program image. exec changes the program running in the existing process context, subject to documented transformations; it does not simply create an unrelated second process. Open descriptors can survive unless configured to close on exec, which matters for accidental credential or socket inheritance.
A service supervisor also controls environment, user identity and resource limits. Reproducing a command in your interactive shell may use different settings from the failing daemon. Inspect the actual process context before concluding that an application error is inconsistent.
Diagnose the symptom
Service supervision adds another boundary: systemd startup and restart behavior determines when a launched process is considered ready and how repeated failures are handled.
Self-check: a service reports “too many open files,” and restarting it clears the error for six hours. What evidence would you collect before the next restart?
Track descriptor counts by type over time, compare them with request and connection volume, inspect the effective process limit, and identify handles that remain after work completes. A steadily growing socket count suggests a lifecycle problem; a stable count near the limit under a legitimate load increase suggests capacity policy needs review. Preserve enough evidence to distinguish those cases.
Use the diagnostic method to choose the next observation that separates your competing explanations.
For a host that reports high load, compare runnable and blocked tasks before deciding the CPUs are saturated.
File descriptor exhaustion follows open files and sockets to their process limit, including a bounded local EMFILE experiment.
Reproduce a lost update in processes, threads and synchronization, then identify the shared invariant a mutex must protect.