Stop the OOM Killer from randomly murdering your database processes

It’s 3:00 AM, and your primary PostgreSQL instance goes down. The logs show Out of memory: Killed process, but the server isn’t under CPU or disk load. You realize the OOM killer picked a process with a high memory footprint (your DB) rather than a memory-hungry batch job that was actually leaking memory, because the kernel’s default scoring algorithm didn’t account for process criticality.

The 5-minute setup

By default, the Linux OOM killer scores processes based on their memory usage relative to the total system memory. This is dangerous for mixed workloads where a critical service (like a database) shares a node with ephemeral batch jobs. You can influence this scoring via /proc/<pid>/oom_score_adj.

To protect critical services, you should adjust their oom_score_adj to a low value (e.g., -500) and allow disposable services to be killed first by setting them to a high value (e.g., 500). Here is how to automate this for a systemd-managed service:

1. Create a drop-in config for your critical service (e.g., postgresql).

    sudo systemctl edit postgresql

1. Add the following to the editor that opens:

    [Service]
    OOMScoreAdjust=-500

1. Now, for your batch job services (e.g., etl-worker), do the same:

    sudo systemctl edit etl-worker

1. Add the following:

    [Service]
    OOMScoreAdjust=500

1. Restart the services to apply changes immediately (or wait for the next start):

    sudo systemctl daemon-reload
    sudo systemctl restart postgresql etl-worker

Why it works

The oom_score_adj file allows you to adjust the OOM score in the range from -1000 to 1000. A score of -1000 effectively disables the OOM killer for that process (it will never be killed), while higher positive values make the kernel prefer that process as a victim when memory is exhausted. By explicitly marking stateful, critical services as low-priority victims and stateless, restartable batch jobs as high-priority victims, you ensure that the kernel sacrifices the "correct" process during a memory crisis, maintaining service availability for your core applications.

Pro Tip

Don’t just set it and forget it. Add a simple monitoring check to ensure your configuration is actually being applied and hasn’t been overridden by a misbehaving init script or container runtime. Run this one-liner in your monitoring agent or a cron job to verify that your critical DB process has the correct score:

# Check if postgresql has the intended OOM score adjustment
pgrep -x postgres | head -1 | xargs -r cat /proc/{}/oom_score_adj 2>/dev/null | grep -q "^-500$" || echo "ALERT: PostgreSQL OOMScoreAdjust is not -500"

This alert will fire if the score is missing, incorrect, or if the process name has changed. It’s a cheap insurance policy against silent configuration drift.