Stop Guessing Kernel Panics: Automate Crash Dump Collection and Analysis

We’ve all had that moment: a production node hard-crashes, the console scrolls by too fast to read, and the machine reboots into a healthy state with no logs in /var/log/syslog to explain why. You are left staring at a blank dmesg buffer, wondering if it was a hardware fault, a bad driver, or a subtle memory corruption bug. Without a captured crash dump, you are debugging blind, relying on vague user reports or trying to reproduce the issue in a non-production environment that behaves differently.

The 5-minute setup

To capture kernel panics automatically on Linux, you need kexec-tools (to load the kernel directly) and kdump (the crash dump service). On modern RHEL/CentOS/Rocky/AlmaLinux systems, kdump is often pre-installed but disabled or misconfigured. On Debian/Ubuntu, you need to install kdump-tools.

Here is the immediate fix for a RHEL-family system to ensure you get a crash dump on the next panic:

# 1. Install necessary tools (if not present)
sudo dnf install -y kexec-tools kdump-utils

# 2. Ensure kdump is enabled and started
sudo systemctl enable kdump
sudo systemctl start kdump

# 3. Verify the reservation of memory for the crash kernel
# This command checks if the crashkernel memory is reserved at boot
sudo kdumpctl status
# Look for: "kdump: reserved crashkernel memory"

# 4. (Optional but recommended) Force a test panic to verify the dump saves
# WARNING: This will crash the machine immediately! Do this on a test node.
echo c > /proc/sysrq-trigger

After the machine reboots, your crash dump will be located in /var/crash/. The structure is typically /var/crash/<timestamp>/vmcore (the raw dump) and vmcore-dmesg.txt (the dmesg log from the crash context).

To analyze it without rebooting or complex tooling, install crash (the GNU crash utility) and point it at the vmlinux debuginfo for your kernel version:

# Install crash utility and debuginfo for the current kernel
sudo dnf install -y crash kernel-debuginfo-$(uname -r)

# Analyze the latest crash dump
# The argument is the path to the vmcore and the vmlinux debug file
sudo crash /usr/lib/debug/lib/modules/$(uname -r)/vmlinux /var/crash/*/vmcore

Inside the crash prompt, you can run standard debugging commands:

crash> bt      # Show backtrace of the crashing task
crash> log     # Show the kernel log buffer from the time of crash
crash> dis -l <address>  # Disassemble the function at the crash address

Why it works

kdump works by reserving a small chunk of physical memory at boot time (usually 256MB–512MB) that is excluded from the main kernel's memory map. When a panic occurs, the kernel uses kexec to jump directly into a pre-loaded, minimal "crash kernel" that runs in that reserved memory. Because the main kernel's memory is still intact in RAM, the crash kernel can copy the entire physical memory image (vmcore) to disk before the power is cut or the system resets. This preserves the exact state of the CPU registers, stack, and memory at the moment of failure, allowing you to reconstruct the call stack and identify the faulty driver or kernel module.

Pro Tip

Don't analyze vmcore files on the production node if you can avoid it. Copy the vmcore and the corresponding vmlinux debuginfo to a dedicated analysis workstation. The vmcore files can be huge (often several GBs). To save disk space and bandwidth, configure kdump to compress the dump automatically by editing /etc/kdump.conf. Add this line to compress the vmcore using LZO (fast) or GZIP (better ratio):

# /etc/kdump.conf
core_collector makedumpfile -c --message-level 1 -d 31

Then, use makedumpfile -l vmcore-dmesg.txt to extract just the dmesg log first. Often, the dmesg log alone is enough to identify the faulting module (e.g., BUG: unable to handle kernel NULL pointer dereference at 0000000000000000 followed by a stack trace pointing to nvidia or xfs). Only dig into the full vmcore with crash if the dmesg is inconclusive.