Stop Guessing: Detect Disk Usage Anomalies with `ncdu` and Baselines

You receive an alert that your /var/log partition is 90% full, but the top-level du command shows only 20% usage. The missing space is likely "deleted but open" files or hidden by permissions issues. Without a historical baseline, you’re manually hunting through directories like a needle in a haystack, risking service downtime while you figure out what’s eating your IOPS.

The 5-minute setup

Instead of relying on static thresholds, establish a dynamic baseline using ncdu (Non-interactive Disk Usage). This tool is faster than du because it reads file system metadata directly and can skip permission-denied errors gracefully.

First, install ncdu if it’s not present (using apt for Debian/Ubuntu):

sudo apt-get update && sudo apt-get install -y ncdu

Create a script to generate a snapshot of your largest files and directories. This script will serve as your baseline. Save it as /usr/local/bin/disk-baseline.sh:

#!/bin/bash
# disk-baseline.sh
# Usage: ./disk-baseline.sh /path/to/mount > /var/backups/disk_baseline_YYYYMMDD.txt

TARGET_DIR="${1:-/var/log}"
TIMESTAMP=$(date +%Y%m%d_%H%M%S)
OUTPUT_FILE="/var/backups/disk_baseline_${TIMESTAMP}.txt"

# Ensure backup directory exists
mkdir -p /var/backups

echo "Generating disk usage snapshot for $TARGET_DIR at $TIMESTAMP"

# Run ncdu in non-interactive mode, outputting JSON for easy parsing later
# -x: don't cross mount points
# -n: skip hidden files (optional, adjust based on needs)
# -o: save the scan to a file for later comparison
ncdu -x -o /tmp/ncdu_scan_${TIMESTAMP}.json "$TARGET_DIR"

# Extract top 20 largest directories/files for human-readable baseline
echo "Top 20 Largest Items in $TARGET_DIR:"
ncdu -x -j /tmp/ncdu_scan_${TIMESTAMP}.json "$TARGET_DIR" 2>/dev/null | \
  jq -r '.[] | select(.size > 0) | [.path, (.size | tostring)] | @tsv' | \
  sort -k2 -nr | head -20 > "$OUTPUT_FILE"

echo "Baseline saved to $OUTPUT_FILE"

Make it executable and run it to create your initial baseline:

chmod +x /usr/local/bin/disk-baseline.sh
/usr/local/bin/disk-baseline.sh /var/log

Now, create a comparison script that alerts if any directory has grown by more than 20% since the last baseline. Save as /usr/local/bin/disk-anomaly-check.sh:

#!/bin/bash
# disk-anomaly-check.sh
# Compares current disk usage against the latest baseline

BASELINE_DIR="/var/backups"
LATEST_BASELINE=$(ls -t $BASELINE_DIR/disk_baseline_*.txt 2>/dev/null | head -1)

if [ -z "$LATEST_BASELINE" ]; then
    echo "No baseline found. Run disk-baseline.sh first."
    exit 1
fi

TARGET_DIR="/var/log"
CURRENT_JSON="/tmp/ncdu_current.json"

# Scan current state
ncdu -x -o $CURRENT_JSON "$TARGET_DIR" > /dev/null 2>&1

# Extract current top items
CURRENT_ITEMS=$(ncdu -x -j $CURRENT_JSON "$TARGET_DIR" 2>/dev/null | \
  jq -r '.[] | select(.size > 0) | [.path, (.size | tostring)] | @tsv' | \
  sort -k2 -nr | head -20)

# Compare
ALERTS=""
while IFS=$'\t' read -r path size; do
    # Get previous size from baseline
    prev_size=$(grep -E "^${path}\t" "$LATEST_BASELINE" | awk -F'\t' '{print $2}')
    
    if [ -z "$prev_size" ]; then
        # New directory/file
        if [ "$size" -gt 1000000 ]; then # > 1MB
            ALERTS="${ALERTS}\nNEW: $path ($size bytes)"
        fi
    else
        # Calculate growth percentage
        growth=$(( (size - prev_size) * 100 / prev_size ))
        if [ "$growth" -gt 20 ]; then
            ALERTS="${ALERTS}\nGROWTH: $path (+${growth}%, ${prev_size} -> $size bytes)"
        fi
    fi
done <<< "$CURRENT_ITEMS"

if [ -n "$ALERTS" ]; then
    echo "DISK ANOMALY DETECTED IN $TARGET_DIR:"
    echo -e "$ALERTS"
    exit 1
else
    echo "No significant anomalies detected in $TARGET_DIR."
    exit 0
fi

Schedule this check via cron to run every hour:

echo "0 * * * * /usr/local/bin/disk-anomaly-check.sh >> /var/log/disk-anomaly.log 2>&1" | crontab -

Why it works

ncdu is significantly faster than du because it uses system calls to read directory entries directly, avoiding the overhead of stat-ing every file individually. By storing a JSON snapshot, you can perform precise byte-level comparisons rather than relying on human-readable units (KB/MB) which introduce rounding errors. This method catches "silent" growth that static 80% alerts miss, especially in log-heavy environments where file rotation might leave stale data.

Pro Tip

If you’re running this on a busy production server, add ionice -c3 to the ncdu command in both scripts to ensure it doesn’t spike I/O wait and degrade application performance. Example: ionice -c3 ncdu -x -o /tmp/ncdu_scan.json /var/log. This sets the I/O priority to idle, meaning it only runs when the disk is otherwise unused.