Stop Guessing: Monitor Disk Write Latency with `iostat` and `pidstat`

It’s 2 AM, and your CI/CD pipeline is timing out. The CPU looks fine, memory is stable, but the build agents are freezing. You suspect the disk, but df -h shows plenty of space, and iostat without the right flags only shows aggregate averages that hide the painful spikes. You need to pinpoint which process is causing the I/O wait and how long writes are actually taking to hit the platter (or SSD).

The 5-minute setup

First, ensure you have the sysstat package installed (it provides iostat and pidstat). On Debian/Ubuntu: sudo apt-get install sysstat. On RHEL/CentOS: sudo dnf install sysstat.

Now, open two terminals. In the first terminal, run this command to monitor disk I/O statistics every 2 seconds. Pay close attention to the w_await (write await) and r_await (read await) columns.

# Monitor disk I/O every 2 seconds
# -x: extended stats (includes await, svctm)
# -t: timestamp
# -y: skip devices with zero activity
iostat -x -t -y 2

Look for any block device (e.g., sda, nvme0n1, vda) where w_await is consistently high (e.g., >10ms for SSDs, >50ms for HDDs) while %util is not necessarily 100%. High await with low util often indicates fragmentation, slow backend storage, or heavy random I/O.

In the second terminal, run this command to correlate I/O wait time to specific processes. This is the key to finding the culprit.

# Monitor process I/O statistics every 2 seconds
# -d: disk I/O statistics
# -w: show I/O wait time (in milliseconds)
# -p ALL: monitor all processes
# -y: skip idle processes (optional, reduces noise)
pidstat -d -w -p ALL 2

Look for processes with high iodelay (I/O wait time). If you see a process like postgres, dockerd, or a custom app spiking in iodelay while your iostat output shows high w_await on the same disk, you’ve found the bottleneck.

Why it works

iostat -x provides device-level metrics, but it doesn’t tell you who is causing the load. pidstat -d provides per-process I/O wait times, but it doesn’t tell you which device is slow. By running them simultaneously and correlating timestamps, you can link a specific process’s I/O wait to a specific device’s high latency. This eliminates the guesswork of "is it the app or the disk?" and helps you decide whether to optimize the application’s I/O patterns, move the workload to a faster volume, or investigate hardware issues.

Pro Tip

If you’re running this on a production system and don’t want to clutter your screen, pipe the output to a file and grep for the worst offenders later. For example:

# Run for 60 seconds, then find processes with >100ms I/O wait
timeout 60 pidstat -d -w -p ALL 2 | awk '$NF > 100 {print}' | sort -u

Also, remember that iostat’s %util is a poor indicator of saturation for multi-queue devices (like NVMe or multi-core spinning disks). Focus on await and svctm instead. A device can have 100% %util but low await if it’s handling many small, fast requests, or low %util but high await if it’s handling few, slow requests. Always trust await over %util for latency diagnostics.