Stop "Out of Memory" Kills by Tuning Cgroup v2 Memory Pressure

You deploy a new version of your application, and within minutes, the oom-killer starts terminating processes. dmesg shows "Out of memory: Killed process," but your monitoring shows the host still has 2GB of free RAM. The issue isn't total memory exhaustion; it's that your container or systemd slice is hitting its hard limit before the kernel has a chance to reclaim memory or trigger swapping, causing a hard crash instead of a graceful degradation or OOM kill of a specific offending thread.

The 5-minute setup

This solution focuses on Systemd units (services), which is the most common scenario for sysadmins managing non-containerized workloads or orchestrating services on a Linux host. If you are using Docker/Kubernetes, note that this principle applies to the container's memory limit, but the tuning mechanism differs (usually via memory.high in cgroups directly or orchestrator settings).

For a systemd service, we will configure soft limits and pressure stall information (PSI) to allow the kernel to handle memory pressure more gracefully.

1. Create a drop-in configuration for your service (replace myapp.service with your actual unit name):

sudo mkdir -p /etc/systemd/system/myapp.service.d
sudo tee /etc/systemd/system/myapp.service.d/memory-tuning.conf <<EOF
[Service]
# Set a hard limit to prevent the service from consuming all system RAM
MemoryMax=2G

# Set a soft limit. When this is reached, the kernel will try to reclaim
# memory from this cgroup before killing processes. This is the key to
# avoiding hard OOM kills.
MemoryHigh=1.5G

# Enable memory pressure stall information (PSI) if not already enabled.
# This helps monitoring tools detect when the service is actually stressed.
# Note: PSI is usually enabled globally via kernel params, but we ensure
# the cgroup hierarchy is set up to utilize it.
EOF

1. Reload systemd and restart the service:

sudo systemctl daemon-reload
sudo systemctl restart myapp.service

1. Verify the settings are applied:

# Check the current memory limits for the service
systemctl show myapp.service -p MemoryMax -p MemoryHigh

# Check real-time memory usage and pressure
# The 'pressure' column shows how much time the cgroup is stalled due to memory pressure
ps -eo pid,comm,psr,%mem --sort=-%mem | head -n 10
cat /sys/fs/cgroup/system.slice/myapp.service/memory.pressure

Why it works

In Cgroup v2, MemoryMax is a hard limit: if the service exceeds it, the OOM killer immediately terminates a process. MemoryHigh is a soft limit: when the service exceeds it, the kernel's memory reclaimer aggressively scans for reclaimable pages in that cgroup. This "throttling" effect slows down the service slightly, allowing the kernel to free up memory or swap it out, rather than immediately killing the process. This gives your application a chance to handle memory pressure internally (e.g., by releasing caches) or degrade gracefully, rather than crashing hard.

Pro Tip

Don't just set MemoryHigh blindly. Use Pressure Stall Information (PSI) to find the right threshold. Run this command to monitor memory pressure over time:

# Monitor memory pressure for the specific service cgroup
# 'avg10' is the average fraction of time the system was stalled due to memory pressure over the last 10 seconds.
watch -n 1 'cat /sys/fs/cgroup/system.slice/myapp.service/memory.pressure | grep "avg10"'

If avg10 stays above 0.1 (10% of the time) for more than a few seconds, your MemoryHigh is too low, and the service is constantly being throttled. Increase MemoryHigh until avg10 drops below 0.05 under normal load, but ensure MemoryMax remains above MemoryHigh to prevent hard OOM kills. This data-driven approach prevents the common mistake of setting limits too tight, which causes latency spikes, or too loose, which risks system-wide instability.