systemd service lifecycle: dependencies, readiness and restart behavior
Understand systemd service types, ordering dependencies and restart limits. Diagnose a service that starts successfully but cannot yet serve requests.
TL;DR: A running process may still be initializing. Choose a service type that reports the startup boundary you need, express dependency ordering separately from activation, and give repeated failures a bounded restart policy.
What “started” tells another service
systemd manages units, with service units describing supervised processes. The unit's startup result is one input to dependency scheduling. It is not a continuous application health check. A daemon can remain alive while its database connections fail or its request queue stops draining.
With Type=simple, systemd considers startup successful very early, before the new process has successfully executed the configured program. Type=exec waits for execution to succeed, which catches failures such as a missing executable. Neither waits for an HTTP listener to become useful. A notification-aware daemon can use Type=notify and send READY=1 after initialization. Its implementation must actually send that notification. These distinctions are documented in the systemd service manual.
For a worker that needs 20 seconds to load its index, “the executable launched” and “the index accepts queries” are different promises. Select the promise deliberately. Do not set Type=notify on an arbitrary executable and expect systemd to infer readiness from its logs.
Ordering and dependency strength answer separate questions
After=database.service orders the units when both participate in a start transaction. It does not, by itself, pull the database into that transaction. Wants= requests another unit with a weaker relationship; Requires= creates a stronger requirement. Combine the relationship with ordering when startup failure must prevent the dependent unit from starting. The exact propagation behavior matters, so consult the unit dependency reference.
Even a correct startup relationship cannot guarantee the dependency stays healthy forever. Applications need bounded retries and reconnection behavior after startup. network-online.target is similarly a boot-time synchronization point whose meaning depends on the network manager; it cannot promise that a remote database remains reachable.
Review a worker's failure policy
This illustrative fragment assumes /opt/reports/worker supports systemd notifications, runs in the foreground and owns its graceful shutdown behavior. It is not an installation command.
[Unit]
Description=Report queue worker
Wants=network-online.target
After=network-online.target
StartLimitIntervalSec=120
StartLimitBurst=4
[Service]
Type=notify
User=report-worker
ExecStart=/opt/reports/worker
Restart=on-failure
RestartSec=5
TimeoutStartSec=45
TimeoutStopSec=30
Suppose a malformed configuration makes the process exit after one second. A five-second delay avoids a tight restart loop, but without a start limit the service can still fail indefinitely. The configured burst caps repeated starts within the interval. Hitting the limit is a diagnostic event; increasing it does not repair the configuration.
A controlled stop is different from an unexpected failure. The worker should stop accepting work, finish or return in-flight jobs, and exit within the stop timeout. If it is killed after that timeout, delivery semantics determine whether the unfinished job is retried. Link the unit's behavior to the queue's delivery guarantees, especially for jobs that charge customers or send notifications.
| Observation | Useful evidence | Likely next investigation |
|---|---|---|
| Start immediately fails | Exec result and journal | Path, permissions, user or configuration |
| Unit active, endpoint unavailable | Listener and readiness timeline | Initialization versus reported startup |
| Repeated short-lived processes | Restart count and exit reason | Crash loop or watchdog failure |
| Stop exceeds its timeout | In-flight work and signal handling | Shutdown deadline and job recovery |
Inspect the effective unit
Read-only commands such as systemctl cat report-worker.service, systemctl show report-worker.service -p MainPID -p Result -p NRestarts and journalctl -u report-worker.service --since '-10 min' connect configuration to runtime evidence. Drop-ins can override the file someone pasted into a ticket. Check the running host's systemd version before relying on a newer directive.
Self-check: the dependent API starts before the worker's index loads, although After=report-worker.service is present. Adding a sleep hides the race only for one observed startup time. First determine the worker's service type and readiness contract. A real readiness notification, or an application-level retry with a deadline, handles variable initialization time without assuming it always finishes in a fixed number of seconds.