Fix the Misleading "No Space Left on Device" from inotify Watch Exhaustion

You're running a monitoring agent that tails thousands of log files, or a developer's file-watching tool (webpack, nodemon, fswatch) starts crashing with OSError: [Errno 28] No space left on device — but df -h shows plenty of disk. This is the inotify subsystem hitting its per-user watch limit, and the error message is a notorious red herring that sends people hunting for disk space instead of a kernel parameter.

The 5-minute setup

First, confirm the problem by checking current vs. max watch counts:

# Count how many inotify watches are currently in use
find /proc/*/fd -lname anon_inode:inotify 2>/dev/null | wc -l

# Check the current limit (default is 8192 on most distros)
cat /proc/sys/fs/inotify/max_user_watches

Then raise the limit persistently:

# Temporarily raise to 524288 (immediate, no reboot)
sudo sysctl -w fs.inotify.max_user_watches=524288

# Also raise the instance and event limits for good measure
sudo sysctl -w fs.inotify.max_user_instances=1024
sudo sysctl -w fs.inotify.max_queued_events=32768

# Make it permanent across reboots
cat <<'EOF' | sudo tee /etc/sysctl.d/99-inotify.conf
fs.inotify.max_user_watches=524288
fs.inotify.max_user_instances=1024
fs.inotify.max_queued_events=32768
EOF

sudo sysctl --system

If you're on a system where the error appears as a generic "Too many open files" instead, the max_user_instances line is the one you actually need — each inotify_init() call consumes one instance.

Why it works

The Linux inotify subsystem tracks filesystem events (create, modify, delete, etc.) by allocating a watch descriptor per directory path. The kernel enforces a per-user cap (max_user_watches) to prevent unprivileged users from exhausting kernel memory. When a process tries to add a watch beyond that cap, the inotify_add_watch() syscall returns ENOSPC (errno 28), which userspace libraries translate to "No space left on device" — the same errno you'd get from an actual disk-full condition. The fix is purely a kernel accounting change; no disk space is involved.

Pro Tip

Audit which processes are hogging watches so you can fix the root cause instead of just raising the ceiling:

# List top inotify consumers by PID
find /proc/*/fd -lname anon_inode:inotify 2>/dev/null |
  cut -d/ -f3 |
  uniq -c |
  sort -rn |
  head -10 |
  while read count pid; do
    echo "$count watches — $(cat /proc/$pid/cmdline 2>/dev/null | tr '\0' ' ')"
  done

If you see a single process holding tens of thousands of watches, it's likely a file-watching loop that isn't cleaning up stale watches (a common bug in older versions of watchdog, nodemon, and some Java-based IDEs). Restarting that specific process frees the watches without needing the sysctl bump — and if you're running a monitoring stack, consider switching from inotify-based tailing to polling (e.g., tail -F uses polling by default) for directories with high churn, since polling avoids the watch limit entirely at the cost of higher CPU.