Stop losing production context: Automate kdump for kernel panics

You’re staring at a black screen. Your critical database node just rebooted itself, and /var/log/syslog shows a generic "system rebooted" message with zero context. You have no idea if it was a memory leak, a faulty NIC driver, or a race condition in a new patch. Without a kernel crash dump, you are left guessing, restarting, and hoping it doesn't happen again during business hours.

The 5-minute setup

Most modern Linux distributions (RHEL/CentOS/AlmaLinux, Ubuntu, Debian) ship with kdump tools, but they are often disabled or misconfigured by default. Here is how to ensure you capture a usable crash dump in under five minutes.

First, verify kdump is installed and enabled. On RHEL-based systems:

sudo dnf install kexec-tools
sudo systemctl enable kdump
sudo systemctl start kdump

On Debian/Ubuntu-based systems:

sudo apt install kdump-tools
sudo systemctl enable kdump-tools
sudo systemctl start kdump-tools

Next, we need to ensure there is enough reserved memory for the crash kernel. If your system has less than 8GB of RAM, the default reservation might fail silently or leave too little space for the live kernel to operate. Check your current reservation:

cat /sys/kernel/kexec_crash_size

If this value is 0, or significantly lower than 256MB, edit the configuration. On RHEL, edit /etc/kdump.conf:

# Reserve at least 256MB for the crash kernel
crashkernel=auto

On Debian/Ubuntu, edit /etc/default/kdump-tools:

# Ensure kdump is enabled
USE_KDUMP=1
# Optionally specify size if auto-detection fails
# KRCD_MEMORY=256M

After changing configs, restart the service to apply the new memory reservation:

sudo systemctl restart kdump

Finally, verify that the kexec payload is loaded. You should see a line indicating the crash kernel is registered:

sudo kdumpctl status

Expected output:

kdump: Crash kernel loaded

Why it works

kdump works by booting a second, minimal "crash kernel" into a reserved block of physical memory at boot time. When the production kernel panics, it doesn't just halt; it triggers kexec, which immediately boots this second kernel. Because this second kernel has its own isolated memory space and doesn't rely on the corrupted state of the first kernel, it can safely write the memory contents of the crashed kernel to disk (usually /var/crash/). This preserves the exact state of all variables, stack traces, and memory structures at the moment of failure.

Pro Tip

Don't let your disk fill up with old dumps. Enable automatic cleanup and compression by adding the following to /etc/kdump.conf (RHEL) or ensuring kdump-tools defaults are set correctly:

path /var/crash
core_collector makedumpfile -c --message-level 1 -d 31
save_kmsg

The -c flag compresses the dump (saving ~70% space), and -d 31 filters out unused pages (like zero-filled memory), creating a "partial dump" that is much smaller and faster to analyze than a full memory image. If you see /var/crash growing unbounded, add a cron job to delete dumps older than 7 days:

echo "0 3 * * * find /var/crash -type d -mtime +7 -exec rm -rf {} +" | sudo tee /etc/cron.d/kdump-cleanup