Prevent Noisy Neighbor OOM Kills with Systemd-OOMD Tuning

You’re on-call at 2 AM because a production web server just went down. The logs show a Killed process message, and you realize a single runaway Java application consumed all available RAM, triggering the kernel’s OOM killer. The OOM killer took out the database process instead of the memory hog, causing cascading failures. The default Linux OOM killer is blunt; it doesn’t know which process is "noisy" or which one is critical. systemd-oomd fixes this by allowing you to define per-service memory limits and kill the specific offending unit before the kernel steps in.

The 5-minute setup

First, ensure systemd-oomd is installed and enabled. On most modern RHEL/CentOS/Debian systems, it is part of systemd.

# Check if oomd is installed
systemctl status systemd-oomd.service

# If not installed (Debian/Ubuntu)
sudo apt install systemd-oomd

# If not installed (RHEL/CentOS)
sudo dnf install systemd-oomd

Next, create a configuration file to define a "high memory" threshold. We want to trigger the kill if any service exceeds 80% of its cgroup memory limit or if the system is under severe pressure. Create /etc/systemd/oomd.conf.d/production.conf:

[oomd]
SwapUsedLimitPercent=80

This means if swap usage exceeds 80%, oomd will start killing processes. But we want to be more precise. Let’s configure a specific service, say my-app.service, to be killed if it exceeds 90% of its assigned memory limit.

Edit your service unit file /etc/systemd/system/my-app.service:

[Service]
# ... existing settings ...
MemoryMax=2G
MemoryHigh=1.8G

MemoryHigh is a "soft" limit. When the service exceeds this, the kernel will reclaim memory aggressively (throttling I/O and CPU) but won’t kill the process immediately. MemoryMax is a "hard" limit. If the service exceeds this, systemd-oomd will kill the process.

Now, we need to tell systemd-oomd to watch this specific service. Create a drop-in file for systemd-oomd: /etc/systemd/system/systemd-oomd.service.d/override.conf:

[Service]
# No specific override needed here, but we ensure it's enabled

Actually, systemd-oomd automatically watches all cgroups that have MemoryHigh or MemoryMax set. However, to explicitly control which services are subject to OOMD (and avoid killing critical system services), you can use the oomd.conf file to exclude certain units.

Let’s refine our approach. Instead of a global swap limit, let’s use the MemoryPressure settings in the service file to give oomd more context.

Update /etc/systemd/system/my-app.service:

[Service]
MemoryMax=2G
MemoryHigh=1.5G
# This tells oomd to kill this service if it's under heavy memory pressure
# and other services are not.
OOMPolicy=kill

OOMPolicy=kill is the default for most non-system services, but it’s good to be explicit. For critical services like sshd or database, you might want OOMPolicy=continue to let the kernel OOM killer handle it (which is less ideal, but sometimes necessary for stability).

Reload systemd and restart the service:

sudo systemctl daemon-reload
sudo systemctl restart my-app.service

To verify that systemd-oomd is active and monitoring, run:

sudo systemctl status systemd-oomd
sudo systemd-oomd

The systemd-oomd command will print the current configuration and the list of monitored cgroups. You should see my-app.service listed with its memory limits.

Why it works

The kernel’s OOM killer makes decisions based on a heuristic score (oom_score) that considers memory usage, process priority, and other factors. It doesn’t understand service boundaries. systemd-oomd operates at the cgroup level. It monitors memory pressure and usage relative to the limits defined in the cgroup. When a cgroup exceeds MemoryHigh, it triggers aggressive memory reclamation. If it exceeds MemoryMax, systemd-oomd sends a SIGKILL to the processes in that cgroup. This ensures that the runaway service is terminated cleanly by the init system, allowing it to be restarted by Restart= directives, rather than having the kernel kill a random process that might be critical to system stability.

Pro Tip

Use journalctl -u systemd-oomd -f to monitor oomd activity in real-time. When oomd kills a process, it logs a clear message like:

systemd-oomd[123]: Killing process 456 (java) of unit my-app.service because memory is under pressure.

This gives you immediate visibility into which service is causing memory issues, making troubleshooting much faster than parsing kernel logs. Additionally, you can set MemoryAccounting=yes in your service files to ensure accurate memory tracking, which is required for oomd to function correctly.