Stop Kernel OOM Kills: Enforce Cgroup Memory Limits with `systemd-oomd`

You deploy a new microservice with a generous memory allocation. A few hours later, the host becomes unresponsive. Checking dmesg, you see the kernel OOM killer has terminated critical system daemons, not just the rogue service. The kernel’s default OOM killer is indiscriminate; it looks at the whole system’s memory pressure and picks a victim based on heuristics, often ignoring cgroup boundaries if the limit isn't set strictly at the unit level or if the cgroup itself isn't monitored by a daemon that reacts faster than the kernel.

The 5-minute setup

First, ensure systemd-oomd is enabled. It is a daemon that watches cgroups for memory pressure and kills processes before the kernel has to intervene globally.

# Enable and start systemd-oomd
sudo systemctl enable --now systemd-oomd

Next, edit your specific service unit file (e.g., /etc/systemd/system/my-app.service) to set a hard memory limit and enable the OOM policy for that specific cgroup.

[Unit]
Description=My Critical Application
After=network.target

[Service]
ExecStart=/usr/bin/my-app --config /etc/my-app.conf

# Hard limit: If this is exceeded, the cgroup is killed immediately
MemoryMax=512M

# Soft limit: Triggers reclaim pressure before hitting Max
MemoryHigh=448M

# Enable the oomd policy for this specific service
# This tells systemd-oomd to monitor this cgroup specifically
OOMPolicy=kill

# Optional: Set a specific score boost (higher = more likely to be killed if system-wide pressure occurs)
# Only use if you want to prioritize killing this over others in a global event
# OOMScoreAdjust=100

[Install]
WantedBy=multi-user.target

Reload the daemon and restart the service:

sudo systemctl daemon-reload
sudo systemctl restart my-app.service

To verify that systemd-oomd is actually monitoring the cgroup, check the status:

systemctl status systemd-oomd
# Look for "Active: active (running)" and ensure no errors in journal

Why it works

The kernel’s OOM killer operates on global system memory pressure. systemd-oomd operates on cgroup memory pressure. By setting MemoryMax and MemoryHigh in the unit file, you define the boundaries for the application’s cgroup. systemd-oomd watches these cgroups and, if the memory usage exceeds MemoryHigh, it triggers a memory reclaim event. If usage hits MemoryMax and the process cannot free memory, systemd-oomd kills the process within that cgroup immediately. This prevents the memory leak from propagating to the rest of the host, ensuring that critical system services (like sshd or dbus) remain untouched because the failure is isolated to the offending cgroup.

Pro Tip

Don’t just set MemoryMax. Always set MemoryHigh to roughly 85-90% of MemoryMax. MemoryHigh triggers the kernel to aggressively swap and reclaim memory before the hard limit is hit, giving your application a chance to survive transient spikes. Without MemoryHigh, your app will go straight from "healthy" to "killed" with no warning. Monitor this with:

# Check current cgroup memory stats for your service
cat /sys/fs/cgroup/system.slice/my-app.service/memory.current
cat /sys/fs/cgroup/system.slice/my-app.service/memory.max

If you see memory.current frequently touching memory.high, your application is under memory pressure. Investigate leaks or increase the limit, rather than waiting for a crash.