Stop the "Zombie" Loop: Auto-Restart Services Killed by OOM Killer

You deploy a memory-hungry Java or Node.js service. Three days later, the monitoring dashboard shows the service is down, but the host is still up. You check dmesg and find the kernel OOM killer terminated the process. Your systemd unit is configured with Restart=always, but because the process was killed by a signal (SIGKILL) from the kernel rather than crashing on its own, systemd often marks it as a "clean" exit or fails to trigger the restart policy correctly, leaving the service dead until a human notices.

The 5-minute setup

First, confirm your systemd unit file (e.g., /etc/systemd/system/my-app.service) has the following in the [Service] section. If Restart is missing or set to no, systemd won't attempt to bring it back up after an OOM kill.

[Service]
ExecStart=/usr/bin/java -jar /opt/app/my-app.jar
Restart=always
RestartSec=5
# Optional: If you want to prevent rapid restart loops if the app is inherently broken
StartLimitIntervalSec=60
StartLimitBurst=5

However, Restart=always alone is not enough to distinguish between a user-initiated stop and an OOM kill. To ensure the service restarts specifically when killed by the OOM killer (and not just any signal), you should rely on systemd's default behavior for Restart=always which covers all exit statuses, but you must ensure the OOM kill is not being ignored by cgroup settings.

A more robust approach for 2026-era systems is to use systemd-run or a wrapper script if you need specific logging, but the native systemd fix is usually simpler. Ensure your cgroup memory limit is set so the OOM killer targets the specific service, not the whole host.

Add this to the [Service] section:

MemoryMax=2G
MemoryHigh=1.8G

Then, reload and restart:

sudo systemctl daemon-reload
sudo systemctl restart my-app.service

To verify this works, simulate an OOM kill. First, check the current memory usage:

systemctl show my-app.service -p MemoryCurrent

Then, manually trigger an OOM kill for testing (use with caution in production):

# Find the PID
PID=$(systemctl show my-app.service -p MainPID --value)
# Kill it with SIGKILL, simulating the kernel's action
sudo kill -9 $PID

Check the status immediately:

systemctl status my-app.service

You should see it as "active (running)" after the RestartSec delay. If it shows "failed," check journalctl -u my-app.service -n 50 for errors.

Why it works

By default, systemd treats a process killed by the kernel (exit code 137, which is 128+9 for SIGKILL) as a failure. Restart=always instructs systemd to restart the unit regardless of the exit status. Setting MemoryMax ensures that when the service exceeds its limit, the kernel's OOM killer is invoked within that specific cgroup context, killing only the offending process rather than potentially selecting a critical system process. This isolation guarantees that the restart policy applies to the correct unit and prevents cascading failures where the OOM killer might terminate a database or SSH daemon instead.

Pro Tip

Don't just rely on restarts; monitor for OOM kills to detect memory leaks before they cause downtime. Add this to your monitoring agent (e.g., Node Exporter, Prometheus, or a simple cron job):

# Count OOM kills for a specific service in the last hour
journalctl -u my-app.service --since "1 hour ago" | grep -c "Killed process.*by signal 9"

If this number is > 0, you have a memory leak or misconfigured limits. Alert on this metric. Also, remember that MemoryHigh triggers a throttling mechanism (the cgroup is slowed down) before MemoryMax triggers an OOM kill. Use MemoryHigh as your "soft limit" for warnings and MemoryMax as your "hard limit" for hard stops.