Stop reacting to "Disk Full" alerts: Implement proactive inode and capacity forecasting

You’re mid-deploy when the build fails with No space left on device, but df -h shows 20% free space. You dig in and discover the filesystem is out of inodes, or that a runaway log file filled the partition in the last hour. By the time you identify the culprit, the service has been down for 15 minutes. Most disk monitoring treats capacity as a static threshold, failing to account for consumption rate or inode exhaustion, which leads to "surprise" outages.

The 5-minute setup

Instead of just checking if usage exceeds 80%, we will set up a lightweight monitoring script that tracks the rate of disk usage increase and inode availability. This script can be run via cron or systemd timer every 5 minutes.

First, create a monitoring script at /usr/local/bin/disk-forecast.sh:

#!/bin/bash
# disk-forecast.sh
# Monitors disk usage rate and inode availability

THRESHOLD_USAGE=80
THRESHOLD_INODES=85
RATE_LIMIT=5 # Alert if usage grows by >5% in the last 15 min

# State file to remember previous usage
STATE_FILE="/var/lib/disk-forecast/state.json"
mkdir -p "$(dirname "$STATE_FILE")"

# Load previous state (empty if new)
PREV_STATE=$(cat "$STATE_FILE" 2>/dev/null || echo "{}")

CURRENT_STATE="{"
FIRST_ENTRY=1

# Iterate over all local filesystems (excluding tmpfs, devtmpfs, etc.)
while read -r filesystem size used avail use_pct mounted_on; do
    # Skip pseudo-filesystems
    if [[ "$filesystem" == /dev/* ]] && [[ "$mounted_on" != /* ]]; then
        continue
    fi
    if [[ "$mounted_on" == /boot/efi ]]; then
        continue
    fi

    # Get current inode usage
    inode_pct=$(df -i "$mounted_on" | awk 'NR==2 {print $5}' | tr -d '%')
    
    # Get current usage percent
    current_usage=$(echo "$use_pct" | tr -d '%')

    # Retrieve previous usage for this mount point
    prev_usage=$(echo "$PREV_STATE" | jq -r --arg mp "$mounted_on" '.[$mp].usage // 0')

    # Calculate rate of change (simple delta for this interval)
    # Note: For a true forecast, you'd need a longer history. 
    # Here we alert if the jump in this interval is suspiciously large.
    delta=$(( current_usage - prev_usage ))

    # Alert conditions:
    # 1. Absolute usage > threshold
    # 2. Inode usage > threshold
    # 3. Sudden spike in usage (delta > RATE_LIMIT)
    
    if [[ $current_usage -ge $THRESHOLD_USAGE ]]; then
        echo "ALERT: High Disk Usage on $mounted_on: ${current_usage}%"
    fi

    if [[ $inode_pct -ge $THRESHOLD_INODES ]]; then
        echo "ALERT: High Inode Usage on $mounted_on: ${inode_pct}%"
    fi

    if [[ $delta -ge $RATE_LIMIT ]]; then
        echo "WARNING: Rapid Disk Growth on $mounted_on: +${delta}% in last interval"
    fi

    # Build new state JSON
    if [[ $FIRST_ENTRY -eq 0 ]]; then
        CURRENT_STATE+=","
    fi
    CURRENT_STATE+="\"$mounted_on\": {\"usage\": $current_usage, \"inode\": $inode_pct}"
    FIRST_ENTRY=0

done < <(df -h --output=source,size,used,avail,pcent,target | tail -n +2)

CURRENT_STATE+="}"

# Save current state
echo "$CURRENT_STATE" | jq . > "$STATE_FILE"

Make it executable and install the dependency:

sudo chmod +x /usr/local/bin/disk-forecast.sh
# Ensure jq is installed
sudo apt-get install jq -y # or yum install jq

Set up a systemd timer to run this every 5 minutes. Create /etc/systemd/system/disk-forecast.service:

[Unit]
Description=Disk Usage Forecasting
After=local-fs.target

[Service]
Type=oneshot
ExecStart=/usr/local/bin/disk-forecast.sh

Create /etc/systemd/system/disk-forecast.timer:

[Unit]
Description=Run Disk Forecasting every 5 minutes

[Timer]
OnBootSec=1min
OnUnitActiveSec=5min
AccuracySec=1min

[Install]
WantedBy=timers.target

Enable and start:

sudo systemctl daemon-reload
sudo systemctl enable --now disk-forecast.timer

Why it works

This approach shifts from static thresholds to dynamic monitoring. By tracking the delta between intervals, it catches runaway processes (like a tail -f piped to disk or a misconfigured backup) before they hit the hard ceiling. The inode check is critical because many modern applications (like Elasticsearch or microservices with many small files) exhaust inodes long before blocks are full, causing cryptic ENOSPC errors that block-based monitoring misses.

Pro Tip

To integrate this with your existing alerting (like PagerDuty or Slack), append an || chain to the echo commands in the script to call curl to your webhook. For example:

if [[ $current_usage -ge $THRESHOLD_USAGE ]]; then
    ALERT_MSG="High Disk Usage on $mounted_on: ${current_usage}%"
    echo "$ALERT_MSG"
    curl -s -X POST -H "Content-Type: application/json" \
         -d "{\"text\":\"$ALERT_MSG\"}" \
         "https://hooks.slack.com/services/YOUR/SLACK/WEBHOOK"
fi

Gotcha: Ensure your state file /var/lib/disk-forecast/state.json is not wiped by tmpwatch or similar cleanup tools, as losing history breaks the rate-of-change calculation.