Stop guessing why your node "hung": Enable Kernel Soft Lockup Detection

You’re on-call when a production node stops responding to SSH and health checks fail, but the machine isn’t dead—the CPU is still at 100%, yet no processes are moving. It’s a classic "soft lockup" where a kernel thread or driver bug has monopolized a CPU core, effectively freezing the system without triggering a hard crash. Without specific kernel logging, you’re left guessing whether it’s a bad disk, a runaway process, or a kernel panic in the making, wasting precious minutes while the service is down.

The 5-minute setup

By default, the kernel’s soft lockup watchdog is often disabled or set to silence non-critical warnings. To ensure you get immediate evidence in your logs the next time this happens, you need to enable the watchdog and configure it to panic (or at least log aggressively) so you can capture a core dump or at least identify the offending task.

First, check the current status. The softlockup_panic parameter determines if a soft lockup should trigger a kernel panic.

# Check if soft lockup detection is currently enabled (1 = enabled, 0 = disabled)
cat /proc/sys/kernel/softlockup_all_cpu_backtrace
cat /proc/sys/kernel/nmi_watchdog

To enable detailed backtraces for all CPUs when a soft lockup is detected (crucial for identifying if it’s a single-core issue or a global freeze), run:

# Enable backtraces for all CPUs on soft lockup
sudo sysctl -w kernel.softlockup_all_cpu_backtrace=1

# Ensure the NMI watchdog is active (required for soft lockup detection on many architectures)
sudo sysctl -w kernel.nmi_watchdog=1

If you want the system to panic immediately upon a soft lockup (so your crash-dumping mechanism, like kdump or kexec, can capture the state), set:

# 1 = Panic on soft lockup, 0 = Just log it
sudo sysctl -w kernel.softlockup_panic=1

To make these changes persistent across reboots, add them to your sysctl configuration file:

sudo tee -a /etc/sysctl.d/99-softlockup.conf <<EOF
kernel.softlockup_all_cpu_backtrace=1
kernel.nmi_watchdog=1
kernel.softlockup_panic=1
EOF

sudo sysctl -p /etc/sysctl.d/99-softlockup.conf

Why it works

The Linux kernel includes a "watchdog" timer that runs on each CPU. If a CPU is stuck in kernel mode (unable to handle interrupts or schedule other tasks) for longer than kernel.watchdog_thresh (default 10 seconds), the watchdog triggers a soft lockup warning. By enabling softlockup_all_cpu_backtrace, the kernel forces a dump of the stack traces for all CPUs, not just the stuck one. This allows you to see if other CPUs are waiting on a lock held by the stuck CPU, providing the exact call stack needed to identify the faulty driver or kernel module.

Pro Tip

If you don’t want the node to panic (because you have critical state in memory), set kernel.softlockup_panic=0 but ensure your log aggregator (like Loki, ELK, or Splunk) is configured to alert specifically on the string "BUG: soft lockup". This string is unique enough to trigger an immediate page, allowing you to correlate the timestamp with other events (like a specific driver update or storage array failure) without losing the running system.