Startup Types, Readiness, Watchdogs and Restarts

A process can exist before it can accept work. A database might still be replaying a log; a web application might be waiting for its configuration. Ordering another service after a process launch is useful only if the manager has a meaningful definition of readiness.

How a service reports startup

Systemd’s Type= selects a startup protocol. With simple, the manager considers the service started after the process is forked, before confirming that the executable ran. exec waits until the executable has been invoked successfully, but does not prove that it has bound a port or can answer a request. oneshot waits for its start command to exit; it is appropriate for bounded work, not a daemon that stays in the foreground. notify expects an application that implements the notification protocol to send READY=1 when it considers itself ready. Legacy daemons may use forking, with their own process-identification considerations.

The precise semantics, timeouts and other types are in the systemd.service manual. Do not switch a program to Type=notify unless it actually sends the required notification. A readiness signal is the application’s assertion; an independent client check can still reveal a broken dependency or wrong response.

Examine the lab service

The bl06-demo Python HTTP server uses Type=exec. On the lab clone:

systemctl show bl06-demo.service \
  -p Type -p ActiveState -p SubState -p MainPID -p Restart -p NRestarts
sudo ss -ltnp 'sport = :18765'
python3 - <<'PYCODE'
import urllib.request
response = urllib.request.urlopen("http://127.0.0.1:18765/", timeout=5)
print(response.status)
PYCODE

The service state and listener are useful evidence; the HTTP response tests an actual request. They are snapshots and can change between commands. Python’s basic server does not send systemd readiness notifications. For a real service, choose a documented startup protocol and test the required dependency and response semantics, including failure and timeouts.

Restart policy and watchdogs

Restart=on-failure requests a restart after qualifying failures; it does not make a failed dependency healthy and does not guarantee an endless retry loop. Start-rate limiting can suppress repeated starts. RestartSec= spaces attempts. Choose a policy with the service’s data integrity and failure mode in mind, and inspect both the service logs and the manager’s restart counters. A deliberately stopped service should not be used as a test of failure recovery.

A watchdog is a deadline for periodic notifications from a cooperating application. WatchdogSec= alone cannot teach an application to send watchdog messages. A process may send heartbeats while a dependent network service or client path is broken, so supplement the watchdog with meaningful health checks. Do not add watchdog settings to the Python lab server.

To inspect restart evidence without injecting a fault:

systemctl show bl06-demo.service -p Restart -p RestartUSec -p NRestarts
sudo journalctl -u bl06-demo.service -b -n 50 --no-pager

A high restart count suggests examining the first failure, not merely resetting the counter. If the unit reaches its start limit, diagnose and correct the cause before reset-failed or another start. The reset clears manager failure/rate-limit state; it does not fix the underlying application.

Worked diagnosis and practice

A dependent unit starts after a web service but its first request fails. Inspect the web unit’s type and startup logs, listener and client response; After= on a Type=exec service provides ordering through execution, not proof that the web server is ready. Use the application’s supported readiness protocol or design retries with bounded timeouts. Record the lab service’s type, restart policy and HTTP response, and explain what each tells you and what it cannot establish.