Stop "No Space Left" Crashes by Monitoring Inodes, Not Just Disk Blocks

You are staring at your dashboard, and the disk usage for /var is only at 40%, yet your application is crashing with ENOSPC (No Space Left on Device) errors. The logs are full of failed writes, but df -h shows plenty of gigabytes free. This is a classic inode exhaustion scenario: the filesystem has run out of inodes (metadata structures) to track new files, even though there is plenty of actual storage space left. This often happens when an application creates millions of small files (like session stores, temp files, or logs) without cleaning them up.

The 5-minute setup

Most standard monitoring agents (like Prometheus node_exporter) report disk usage in bytes, but they often fail to alert on inode exhaustion unless explicitly configured. Here is how to set up a robust check using df -i and a simple systemd service or cron job that alerts you via your preferred channel (e.g., Slack, PagerDuty, or email) before the system goes down.

First, verify the current state on your target host. If Use% is above 90%, you are in danger.

# Check inode usage on all mounted filesystems
df -i

Now, create a monitoring script that parses this output and triggers an alert if the usage exceeds a threshold (e.g., 85%).

Create a file at /usr/local/bin/check_inodes.sh:

#!/bin/bash
# check_inodes.sh - Monitor inode usage and alert if threshold exceeded
THRESHOLD=85
ALERT_CMD="curl -X POST -H 'Content-Type: application/json' -d '{\"text\":\"ALERT: Inode usage on ${HOSTNAME} exceeds ${THRESHOLD}%\"}' https://hooks.slack.com/services/YOUR/SLACK/WEBHOOK"

# Get hostname for context
HOSTNAME=$(hostname)

# Parse df -i output, skip the header line
df -i | tail -n +2 | while read -r Filesystem Inodes IUsed IFree IUse% MountedOn; do
    # Extract the percentage number (strip the %)
    usage=$(echo "$IUse%" | sed 's/%//')
    
    # Check if usage is an integer and exceeds threshold
    if [[ "$usage" =~ ^[0-9]+$ ]] && [ "$usage" -ge "$THRESHOLD" ]; then
        # Log the specific mount point causing the issue
        echo "CRITICAL: Inode usage on ${MountedOn} is ${usage}%" | logger -t inode-monitor
        
        # Send alert (replace with your actual alerting mechanism)
        # For a real setup, ensure this script has network access
        ${ALERT_CMD}
    fi
done

exit 0

Make it executable:

chmod +x /usr/local/bin/check_inodes.sh

Now, schedule it to run every 15 minutes using cron. This is lightweight and doesn't require a dedicated daemon.

crontab -e

Add this line to the crontab:

*/15 * * * * /usr/local/bin/check_inodes.sh >> /var/log/inode_monitor.log 2>&1

If you prefer a systemd approach for more robust logging and dependency management, create a timer unit. Create /etc/systemd/system/inode-monitor.service:

[Unit]
Description=Monitor Inode Usage
After=network.target

[Service]
Type=oneshot
ExecStart=/usr/local/bin/check_inodes.sh
User=root

And /etc/systemd/system/inode-monitor.timer:

[Unit]
Description=Run inode monitor every 15 minutes

[Timer]
OnBootSec=5min
OnUnitActiveSec=15min
AccuracySec=1m

[Install]
WantedBy=timers.target

Enable and start the timer:

systemctl daemon-reload
systemctl enable --now inode-monitor.timer

Why it works

Filesystems allocate inodes when they are created. Each inode contains metadata (permissions, ownership, size, pointers to data blocks) but not the data itself. When you fill up the inode table, the filesystem cannot create new files, directories, or hard links, even if there are terabytes of free space. Monitoring df -i (inode usage) instead of just df -h (block usage) catches this specific failure mode. The script parses the IUse% column, which is the most reliable indicator of risk, and triggers an alert before the 100% mark is reached, giving you time to clean up small files or remount with a different filesystem configuration.

Pro Tip

If you are running a high-volume application (like a message queue or web server) that creates many small files, consider switching your storage backend to a filesystem with a higher inode ratio (like XFS with inode size=256 vs. ext4) or, better yet, redesign the application to use a database or object storage instead of the local filesystem for small file persistence. To quickly identify which directory is eating up inodes, use:

# Find the directory with the most files (inodes) in /var
find /var -type f | cut -d'/' -f2 | sort | uniq -c | sort -nr | head -n 5