"My Service Just Vanished" — Diagnosing and Surviving the OOM Killer

Your Java microservice has been running fine for three weeks. At 3:47 AM, systemd reports the unit exited with status 9 (SIGKILL). No application error in your logs. The process simply ceased to exist. This is the OOM killer — the kernel's last-resort mechanism that selects and kills a process when memory is critically exhausted. It fires silently, leaves almost no trace in application logs, and often picks the wrong victim because it scores processes by RSS size, not by criticality.

The 5-minute setup

First, confirm the OOM killer was responsible and identify what it killed:

# Search kernel ring buffer for OOM events (last 24h)
journalctl -k --since "24 hours ago" | grep -i "oom\|killed process"

# Example output:
# Out of memory: Killed process 14832 (java) total-vm:8291456kB,
# anon-rss:4194304kB, file-rss:0kB, shmem-rss:0kB

# Check if a specific PID was the OOM victim
grep -r "Killed process 14832" /var/log/kern.log

Now, understand why that process was chosen. The kernel assigns each process an oom_score (0–1000, higher = more likely to be killed):

# Read the current OOM score for a running process (e.g., your DB)
cat /proc/$(pidof postgres)/oom_score

# Read the OOM score adjustment you can set (-1000 to +1000)
cat /proc/$(pidof postgres)/oom_score_adj

To protect critical processes from being the OOM victim, set their oom_score_adj to a negative value. Add this to your systemd unit file under [Service]:

# /etc/systemd/system/postgres.service.d/oom-protection.conf
[Service]
OOMScoreAdjust=-500

Then reload and restart:

systemctl daemon-reload
systemctl restart postgres

Verify it took effect:

cat /proc/$(pidof postgres)/oom_score_adj
# Should output: -500

To make the OOM killer prefer killing a specific noisy process (like a batch job or cache), give it a positive adjustment:

[Service]
OOMScoreAdjust=800

Why it works

The OOM killer walks the process list under memory pressure, computes a score based on RSS, adjusts it by oom_score_adj, and kills the highest-scoring process with SIGKILL. Setting oom_score_adj to -1000 makes a process immune (the kernel will not select it), while +1000 makes it the first candidate. This is your only reliable way to control which process dies when memory runs out — the kernel does not care about your SLAs or service dependencies.

If you're running inside a cgroup (Docker, systemd slices, Kubernetes), the OOM kill is scoped to that cgroup's memory limit. Check the cgroup-level OOM count:

# cgroup v2 (modern systems)
cat /sys/fs/cgroup/system.slice/postgres.service/memory.events | grep oom
# memory.events: low=0 high=0 max=0 oom=1 oom_kill=1

# cgroup v1 (older)
cat /sys/fs/cgroup/memory/system.slice/postgres.service/memory.oom_control

Pro Tip

Set up a one-liner alert that fires the moment an OOM kill is logged, so you find out before your users do:

# Add to rsyslog or use journald forwarding — but for a quick check:
journalctl -k --since "5 minutes ago" | grep -q "Killed process" && \
  echo "[$(date)] OOM kill detected" | logger -t oom-alert

Or better, use a systemd path unit watching kern.log with an inotify trigger that sends to PagerDuty. The real gotcha: if you're on cgroup v2 and your container's memory limit is hit, the OOM kill only affects that cgroup — your host dmesg output will look clean. Always check memory.events inside the container's cgroup path, not just the global kernel log.