systemd Service Hardening — Stop Failed Services From Crashing Your Servers

🛠️ DEVOPS TIP — July 17, 2026

systemd Service Hardening — Stop Failed Services From Crashing Your Servers

The Problem: Services configured with Restart=always will loop infinitely on failure, consuming CPU, filling logs, and potentially starving other services — all silently.

The Fix — 5 Hardening Steps:

1. Set restart limits

[Service]
StartLimitBurst=5
StartLimitIntervalSec=60

Stops restart loops after 5 failures in 60 seconds.

2. Add resource limits via cgroup

MemoryMax=2G
MemoryHigh=1.5G
CPUQuota=400%

Prevents a single service from consuming all server resources.

3. Pre-start health checks

ExecStartPre=/usr/bin/test -f /run/db-ready
ExecStartPre=/bin/sleep 2

Ensures dependencies are actually ready before starting.

4. Alert on failure

ExecStopPost=/usr/bin/curl -X POST http://alerts.internal/webhook

Get notified when something goes wrong.

5. Namespace isolation

ProtectSystem=strict
ProtectHome=yes
NoNewPrivileges=yes
PrivateTmp=yes

Limits blast radius if the service is compromised.

💡 Pro Tip: Use drop-in overrides instead of editing unit files directly:

systemctl edit myservice  # creates /etc/systemd/system/myservice.service.d/override.conf
systemd-delta  # audit all overrides
systemd-analyze blame  # debug slow boot