Stop "No Memory" Crashes by Using Cgroup Limits Instead of ulimit

A junior sysadmin sets ulimit -m 1024 on a Java application to prevent it from consuming all RAM, but the app still OOM-kills the host because the kernel ignores virtual memory limits for heap allocations. The resulting outage reveals that traditional POSIX resource limits are ineffective for modern managed runtimes that pre-allocate large virtual address spaces.

The 5-minute setup

Stop relying on ulimit for memory control. Instead, define a dedicated systemd service unit with explicit memory limits using cgroups v2. This ensures the kernel enforces hard caps on physical memory usage, triggering a graceful OOM kill of the process group rather than the entire host.

Create a file at /etc/systemd/system/myapp.service:

[Unit]
Description=My Critical Application
After=network.target

[Service]
ExecStart=/opt/myapp/bin/start.sh
Restart=on-failure
RestartSec=5s

# --- Memory Limits ---
# Hard limit: 2GB. If exceeded, kernel kills the process.
MemoryMax=2G

# Soft limit: 1.8GB. Triggers OOM killer if soft limit is hit 
# and memory is available, providing a buffer before hard limit.
MemoryHigh=1.8G

# Optional: Limit memory for swap as well
MemorySwapMax=512M

[Install]
WantedBy=multi-user.target

Reload systemd and start the service:

sudo systemctl daemon-reload
sudo systemctl enable --now myapp

To verify the limits are applied correctly, check the cgroup memory events:

# Check current usage vs limits
cat /sys/fs/cgroup/system.slice/myapp.service/memory.current
cat /sys/fs/cgroup/system.slice/myapp.service/memory.max

# Monitor for OOM events (incrementing count means limit was hit)
cat /sys/fs/cgroup/system.slice/myapp.service/memory.events

Why it works

ulimit -m (RLIMIT_AS) restricts the virtual address space, which is often much larger than physical RAM usage due to memory-mapped files, shared libraries, and pre-allocated heaps in JVMs or Go runtimes. The kernel does not enforce physical memory usage via ulimit. In contrast, systemd creates a dedicated cgroup (Control Group) for the service. Cgroups operate at the kernel level, tracking actual physical page allocations. When MemoryMax is exceeded, the kernel’s OOM killer targets the specific cgroup, killing only the offending process group rather than selecting a victim from the entire system, thus preserving host stability.

Pro Tip

Don’t just set MemoryMax. Always set MemoryHigh as well. MemoryHigh acts as a "throttling" threshold. When usage exceeds MemoryHigh, the kernel aggressively reclaims memory from that cgroup and throttles its CPU allocation to force it to free pages. This gives your application a chance to handle memory pressure gracefully (e.g., via GC or cache eviction) before hitting the hard MemoryMax limit, which results in an immediate, ungraceful kill. Monitor memory.events periodically; if high events are frequent, your MemoryHigh is too low for the workload.