Prevent "No Space Left on Device" Crashes Caused by Inode Exhaustion

You have 50GB of free disk space on your application server, yet your web service suddenly starts returning 500 errors, and df -h shows plenty of room. The real culprit is inode exhaustion: your filesystem has run out of inodes (metadata structures for files), even though the data blocks are free. This often happens due to a bug creating millions of tiny files (like session tokens or cache fragments) or a runaway cron job failing to clean up.

The 5-minute setup

First, diagnose the issue to confirm it's inodes, not blocks. Run df -i instead of df -h. If the IUse% column is at 100% (or very close), you have an inode problem.

# Check inode usage per filesystem
df -i

# Example output:
# Filesystem      Inodes  IUsed   IFree IUse% Mounted on
# /dev/sda1     1048576 1048576       0  100% /

To find which directory is consuming the most inodes, use this one-liner to count files per top-level directory. This is faster than find on a massive tree because it stops at the first level and counts efficiently.

# Count files in each immediate subdirectory of /
sudo find / -xdev -maxdepth 2 -type f | cut -d'/' -f2 | sort | uniq -c | sort -nr | head -10

If you suspect a specific service (e.g., /var/log or /tmp), drill down deeper:

# Find the top 10 directories inside /var/log by file count
sudo find /var/log -xdev -type f | cut -d'/' -f3 | sort | uniq -c | sort -nr | head -10

Once identified, clean up the offending files. For example, if /var/cache/app is full of stale session files:

# Remove files older than 1 day in a specific cache directory
sudo find /var/cache/app -type f -mtime +1 -delete

To prevent this in the future, add a proactive monitoring check. Create a systemd service or a cron job that alerts you when inode usage exceeds 90%.

# Create /usr/local/bin/check-inodes.sh
#!/bin/bash
THRESHOLD=90
# Get the IUse% for the root filesystem (or specify /dev/sda1)
IUSE=$(df -i / | awk 'NR==2 {print $5}' | sed 's/%//')

if [ "$IUSE" -ge "$THRESHOLD" ]; then
    # Send an alert via your preferred method (email, Slack webhook, etc.)
    echo "ALERT: Inode usage on / is at ${IUSE}%" | mail -s "Disk Inode Warning" admin@example.com
    # Or log to syslog
    logger -t inode-monitor "WARNING: Inode usage on / is at ${IUSE}%"
fi

Make it executable and schedule it:

sudo chmod +x /usr/local/bin/check-inodes.sh
# Add to root's crontab (runs every 5 minutes)
sudo crontab -e
# Add this line:
*/5 * * * * /usr/local/bin/check-inodes.sh

Why it works

Filesystems allocate inodes at creation time based on the ratio of inodes to data blocks (set during mkfs). Each file, directory, and symlink consumes one inode, regardless of its size. When inodes are exhausted, the kernel cannot create new files, even if there is ample free space on disk. This is a common failure mode for high-throughput services that generate many small files without proper cleanup. By monitoring IUse% and identifying the source directory, you can pinpoint the leak and implement cleanup or adjust filesystem parameters.

Pro Tip

If you frequently run into inode limits on modern SSDs, consider reformatting your filesystem with a higher inode ratio. For ext4, you can set the bytes-per-inode ratio during formatting. The default is often 16384 bytes per inode. For workloads with many small files, you might want a lower ratio (more inodes).

# Example: Format with 4096 bytes per inode (4x the default inodes)
# WARNING: This erases all data on the device!
sudo mkfs.ext4 -i 4096 /dev/sdb1

Alternatively, if you cannot reformat, you can monitor the ratio and plan for expansion. Use tune2fs -l /dev/sda1 | grep "Inode" to check your current inode count and block size. If you're consistently hitting inode limits, it's a sign that your application's file-handling strategy needs review—consider consolidating small files into larger ones or using a database/cache system that handles metadata more efficiently.