Stop chasing ghost processes: Detect silent memory leaks in systemd-managed services before they OOM-kill your node

You’re staring at a Grafana dashboard showing a steady, linear climb in node memory usage over the last 48 hours. top shows no single process spiking, and the OOM killer hasn’t fired yet. The culprit is likely a long-running systemd service that is slowly leaking memory but isn’t hitting its MemoryMax limit because it’s fragmented across multiple cgroups or the limit is set too high. By the time the node actually crashes, you’re hours into a post-mortem instead of minutes.

The 5-minute setup

First, identify the top memory consumers within the cgroup hierarchy to isolate the leaking service. This is more accurate than ps because it accounts for kernel memory associated with the service’s cgroup:

# List the top 5 services by current memory usage (in human-readable format)
systemd-cgtop --memory -n 5

# Drill down into a specific service (e.g., 'my-leaky-app.service') to see its cgroup details
systemctl status my-leaky-app.service

Once you’ve identified the service, enable detailed memory accounting and set a hard limit that will trigger an alert (or kill) before the entire node runs out of RAM. Edit the service’s override file:

systemctl edit my-leaky-app.service

Paste the following configuration. Replace 2G with a value appropriate for your service (leave headroom for the OS and other services):

[Service]
# Enable detailed memory accounting to track peak and current usage accurately
MemoryAccounting=yes

# Set a hard limit. If exceeded, the kernel will OOM-kill the service,
# which is better than letting the whole node die.
MemoryMax=2G

# Optional: Set a soft limit to trigger a warning in logs before the hard limit
MemoryHigh=1.5G

# Ensure the service is restarted automatically after an OOM kill
Restart=on-failure
RestartSec=5s

Save and exit, then reload and restart the service:

systemctl daemon-reload
systemctl restart my-leaky-app.service

To verify the limits are active, check the cgroup memory files directly:

# Check the current memory limit for the service's cgroup
cat /sys/fs/cgroup/system.slice/my-leaky-app.service/memory.max

# Check the current usage
cat /sys/fs/cgroup/system.slice/my-leaky-app.service/memory.current

Why it works

systemd services run in isolated cgroups. By default, memory accounting is often coarse or disabled for detailed metrics, making it hard to distinguish between a service that is legitimately using memory and one that is leaking. Enabling MemoryAccounting=yes ensures that the kernel tracks memory usage per cgroup accurately. Setting MemoryMax creates a hard boundary: if the service tries to allocate beyond this limit, the kernel’s OOM killer will terminate the service process immediately. This converts a slow, node-wide degradation into a fast, contained service failure that systemd can automatically restart, preserving overall system stability.

Pro Tip

Don’t just wait for the OOM kill. Set up a Prometheus node exporter alert or a simple cron job to monitor memory.high violations. You can check if a service has hit its soft limit (which logs a warning but doesn’t kill) by inspecting the cgroup’s memory.events file:

# Watch for low-memory pressure events in the service's cgroup
watch -n 1 'cat /sys/fs/cgroup/system.slice/my-leaky-app.service/memory.events'

If you see high or max counters incrementing, the service is hitting its limits. This gives you early warning to investigate the leak before the service is killed, allowing you to capture heap dumps or debug logs while the process is still alive and in a reproducible state.