Stop "System Error: No space left on device" from Inotify Watch Exhaustion

You’re monitoring a high-traffic CI/CD runner or a file-syncing agent like rsync or unison. Suddenly, the service crashes with ENOSPC (No space left on device), even though df -h shows plenty of free disk space. You check dmesg and see nothing about disk capacity, just the application failing to create new file watchers. This is a classic silent killer: the kernel’s inotify subsystem has hit its per-user or system-wide limit, preventing the creation of new watchers for file changes.

The 5-minute setup

First, confirm the issue. Check the current usage against the limit:

# Check current number of inotify instances (watchers)
cat /proc/sys/fs/inotify/max_user_instances

# Check the limit for how many instances a user can create
cat /proc/sys/fs/inotify/max_user_watches

If your application is failing, the current count is likely equal to or exceeding max_user_watches. To fix this immediately without a reboot, update the system-wide limits. For a production server with heavy file I/O (like a build node), increasing these values is safe and standard practice.

# Increase the maximum number of file watches per user
sudo sysctl -w fs.inotify.max_user_watches=524288

# Increase the maximum number of inotify instances per user
sudo sysctl -w fs.inotify.max_user_instances=8192

To make this persistent across reboots, add the following lines to /etc/sysctl.d/99-inotify.conf:

fs.inotify.max_user_watches = 524288
fs.inotify.max_user_instances = 8192

If you are running a specific application that requires an even higher limit (e.g., monitoring a directory tree with 100,000+ files), you can also set a per-user limit if you are using ulimit in a service manager like systemd. In your .service file under [Service]:

LimitNOFILE=65536
# Note: inotify limits are system-wide via sysctl, but ensure the process has enough file descriptors to hold the watchers

Why it works

The Linux kernel uses inotify to notify userspace programs of filesystem events. Each watched file or directory consumes a kernel resource called a "watch." By default, the limit is often set conservatively (e.g., 8192 or 65536) to prevent a buggy or malicious application from exhausting kernel memory and causing a system-wide DoS. When an application needs to watch many files (like a build system watching source code, or a sync tool watching a mounted share), it quickly hits this ceiling. Increasing the sysctl values tells the kernel it’s safe to allocate more memory for these watchers, resolving the ENOSPC error that is actually a resource limit, not a disk space issue.

Pro Tip

Don’t just crank the numbers blindly. Before increasing the limit, verify if your application is leaking watchers. Run this command to see which process is consuming the most inotify resources:

# Find the process with the most open inotify instances
lsof -nP | grep -i inotify | awk '{print $2}' | sort | uniq -c | sort -rn | head

If a single process is holding thousands of watchers, it might be a bug in your application (e.g., adding a new watcher on every file change without removing the old one). Fixing the leak is better than just raising the limit, which will eventually be hit again. If you must raise the limit, monitor fs.inotify.max_user_watches usage via Prometheus or node_exporter to alert before you hit the ceiling again.