3. Missing Dependencies at Runtime

🛠️ DevOps Tip — Aug 17, 2026

systemd Service Reliability — Stop Guessing Why Your Service Keeps Restarting

Your service crashes at 3 AM. systemctl status says "exit code 1" but doesn't explain why. Here are the 5 hidden failure modes and how to debug them:

1. OOM Killer — Service killed by the kernel

dmesg -T | grep -i oom
journalctl -u myservice --no-pager -n 500 | grep -i "killed process"

2. File Descriptor Limit — "Too many open files"

systemctl show myservice --property=LimitNOFILE
cat /proc/$(pgrep myservice)/limits | grep "open files"

3. Missing Dependencies at Runtime

ldd /path/to/binary | grep "not found"
journalctl -u myservice --no-pager | grep -i "cannot open"

4. Port Already in Use

ss -tlnp | grep :PORT

5. Misconfigured Restart Policy — Service restarts too fast, hits systemd's start limit

systemctl show myservice --property=StartLimitIntervalSec,StartLimitBurst

🔧 Hardened Unit Template:

[Service]
Restart=on-failure
RestartSec=5s
StartLimitIntervalSec=120
StartLimitBurst=5
MemoryMax=2G
LimitNOFILE=65536
WatchdogSec=30s
ProtectSystem=strict
ProtectHome=read-only

💡 Golden Rule: journalctl is the diagnosis. systemctl status is just the symptom. Always dig deeper.

📋 Pro Tip: Set up systemd-analyze blame on boot and add a cron that logs service restart counts so you catch degradation before it becomes an outage.