Automating server restarts and health checks with systemd services

A systemd health check closes the gap between a process that is running and a service that is actually working. Restart=on-failure only helps when a process exits — it does nothing for a worker that is stuck, deadlocked or answering every request with a 500. This guide wires up three layers on a Linux VPS: a restart policy for crashes, a timer that runs a health check script and restarts the service after repeated failures, and a watchdog for services that can report their own status.

Before you start

You need:

• A Linux VPS with root or sudo access. Run systemctl --version to confirm systemd is present (any Ubuntu 22.04/24.04, Debian 12, AlmaLinux 8–9 or Rocky 8–9 image qualifies).
• A service unit you control — this guide uses myapp.service as the example. Replace it with your real unit name.
• curl installed for the HTTP check. On minimal images run apt install curl or dnf install curl.
• A health endpoint that reflects real dependencies, not just "the web server answered". A route that touches the database and cache is far more useful than a static file.

Understand what systemd can and cannot detect

systemd watches process state, not application state. If your process exits with a non-zero code, systemd sees it. If it hangs forever, systemd sees a healthy unit. That is why you need both a restart policy and an external check.

Add this to your unit in /etc/systemd/system/myapp.service:

[Unit]
Description=My application
StartLimitIntervalSec=300
StartLimitBurst=5

[Service]
ExecStart=/usr/local/bin/myapp
Restart=on-failure
RestartSec=5s

Restart=on-failure restarts the process after a crash or a non-zero exit. RestartSec=5s waits five seconds so a fast crash loop does not hammer the CPU. StartLimitBurst=5 with StartLimitIntervalSec=300 means five restarts inside five minutes; after that systemd gives up and marks the unit failed, which is exactly what you want — a permanently broken service should stop flapping and page you instead.

If you want a notification when that happens, add OnFailure=myapp-alert.service to the [Unit] section and create a small oneshot unit that sends a message to your monitoring or ticket system.

Write a health check script

The script below checks an HTTP endpoint, counts consecutive failures in a state file, and restarts the service once the count crosses a threshold. Create it at /usr/local/bin/myapp-healthcheck.sh:

#!/usr/bin/env bash
set -euo pipefail

SERVICE="myapp.service"
URL="http://127.0.0.1:8080/healthz"
STATE_DIR="/var/lib/healthcheck"
FAIL_FILE="${STATE_DIR}/${SERVICE}.fails"
MAX_FAILS=3

mkdir -p "$STATE_DIR"

if curl -fsS --max-time 5 "$URL" >/dev/null; then
  rm -f "$FAIL_FILE"
  exit 0
fi

fails=$(cat "$FAIL_FILE" 2>/dev/null || echo 0)
fails=$((fails + 1))
echo "$fails" > "$FAIL_FILE"

if [ "$fails" -ge "$MAX_FAILS" ]; then
  logger -t healthcheck "restarting $SERVICE after $fails failed checks"
  systemctl restart "$SERVICE"
  rm -f "$FAIL_FILE"
fi

exit 1

Make it executable:

chmod 755 /usr/local/bin/myapp-healthcheck.sh

The counter matters. A single failed request during a deploy or a slow query should not trigger a restart, so the script waits for three consecutive failures. curl -fsS returns non-zero on connection refused, timeouts and HTTP 4xx/5xx responses, which covers most failure modes.

Run the check on a systemd timer

Two units are needed: one that runs the script, one that schedules it. Create /etc/systemd/system/myapp-healthcheck.service:

[Unit]
Description=Health check for myapp

[Service]
Type=oneshot
ExecStart=/usr/local/bin/myapp-healthcheck.sh

Then /etc/systemd/system/myapp-healthcheck.timer:

[Unit]
Description=Run the myapp health check every minute

[Timer]
OnBootSec=2min
OnUnitActiveSec=1min
AccuracySec=5s
Unit=myapp-healthcheck.service

[Install]
WantedBy=timers.target

OnBootSec=2min gives the service time to start after a reboot before the first check runs. OnUnitActiveSec=1min schedules the next run one minute after the previous one finished, so a slow check never overlaps itself. AccuracySec=5s keeps the timer from waking the CPU at an arbitrary moment.

Enable and start it:

systemctl daemon-reload
systemctl enable --now myapp-healthcheck.timer

Use a watchdog for in-process health

A timer can only see the service from the outside. If your application can report its own status, WatchdogSec is faster and more accurate — systemd kills and restarts the unit when the heartbeat stops.

[Service]
Type=notify
NotifyAccess=main
WatchdogSec=30s
Restart=on-failure

The process must send READY=1 when it finishes starting and WATCHDOG=1 at least every 15 seconds (half the watchdog interval). A shell example:

systemd-notify --ready
while true; do
  systemd-notify WATCHDOG=1
  sleep 10
done

Most application servers and frameworks have a native integration, so check your runtime's documentation rather than shelling out to systemd-notify. If the heartbeat stops, systemd logs Watchdog timeout and restarts the unit.

Check it works

Validate the unit syntax before reloading:

systemd-analyze verify /etc/systemd/system/myapp-healthcheck.timer
systemctl daemon-reload
systemctl list-timers --all | grep myapp

You should see the timer with a NEXT time in the near future. Run the script by hand to confirm the exit code:

/usr/local/bin/myapp-healthcheck.sh; echo "exit: $?"

Then simulate a failure. Stop the application without disabling it, wait three minutes, and watch the journal:

systemctl stop myapp
journalctl -u myapp-healthcheck.service -n 50 --no-pager
systemctl status myapp

You should see three failed checks, a restarting myapp.service line from logger, and the unit back in active (running) state.

Troubleshooting

• The timer never fires. The [Install] section is missing or the timer was not enabled. Run systemctl enable --now myapp-healthcheck.timer and confirm with systemctl list-timers --all.
• The service restarts in a loop, then stops. StartLimitBurst was reached and systemd marked the unit failed. Read journalctl -u myapp -n 100, fix the underlying fault, then systemctl reset-failed myapp before starting it again.
• The check passes but users still see errors. The endpoint is too shallow. Make it query the database and cache, or check a real request path instead of a static file.
• systemd-notify: No medium found. The unit is not Type=notify, or NotifyAccess does not cover the process sending the message. Set NotifyAccess=all while debugging, then tighten it.
• Permission denied writing the state file. The script runs as a user that cannot write /var/lib/healthcheck. Either run it as root or add StateDirectory=healthcheck to the service unit and use /var/lib/healthcheck as the path.
• curl: command not found. Install curl, or replace the probe with ss -ltn "sport =:8080" to test only that the port is listening.

Next steps

• Deploy a Python app with Gunicorn, systemd and Nginx — a full unit file you can add these policies to.
• Automating server maintenance with cron jobs and shell scripting — schedule log rotation and cleanup alongside your health checks.
• Add health checks to Docker containers — the same idea for containerised workloads.