First: CPU-bound or IO-bound
A quick discriminator: compare load average against CPU utilization in top. If load is high (say 8) while %Cpu(s) us+sy is far below 100% (say 40%), many processes sit in uninterruptible sleep (D state) waiting on IO. The D-state processes in top are the direct evidence.
Better evidence is /proc/pressure/io. Read the some avg10 line: a value near or above 10 means that over the last 10 seconds, an average of 10% of wall time had at least one task stalled on IO. That number tracks perceived sluggishness far better than any single device metric.
High load can also come from non-D artifacts — zombie processes, a fork storm inside containers. So order the checks: count D-state processes, then PSI, then iostat. Only when all three agree should you conclude IO-bound.
If PSI io some is low while load stays high, find the load source: ps -eo state,pid,comm | grep -c "^D" for an exact count, then vmstat 1 and watch the b column (tasks blocked on IO) alongside wa. A persistently positive b with high wa confirms IO wait rather than CPU contention.
ps -eo state,pid,comm | awk '$1=="D"' | head
grep -E "some|full" /proc/pressure/io
vmstat 1 5 # 关注 wa（iowait）与 b（阻塞数）Reading iostat -x column by column
%util only means "fraction of the sampling window during which the device was not idle". For block devices it is the fraction of time with at least one request in flight; it says nothing about how deep the queue is. An NVMe can sit at %util 100% with perfectly healthy throughput and latency — normal behavior for a parallel device, not a fault signal.
The columns that matter are await and the queue-related ones. Modern kernels split r_await / w_await in iostat -x. %aqu-sz and %arq-sz are the average queue wait and average queue size for read requests — rising values mean requests are piling up in front of the device.
Diagnose by combination. High await with low %arq-sz means the device itself is slow: replace the disk or change storage class. High await with high %arq-sz means queue congestion, typically too much concurrency or random IO saturating queue depth. Low await with high %util means many small requests overwhelming parallelism — the classic small-file storm.
Get the sampling interval right. With iostat -x 1 the first row is the since-boot average and is useless — skip it and read from the second row onward. Use iostat -x 1 10 for ten samples, or iostat -x 0.5 20 when you suspect momentary spikes.
iostat -x 1 10
# 关注列：r_await/w_await（平均延迟，含排队）
# %arq-sz / %aqu-sz（请求平均队列深度 / 排队时长）
# rkB/s、wkB/s（吞吐）、%util（只是非空占比）PSI: turning "the device is slow" into "who is being slowed"
PSI (Pressure Stall Information) measures the fraction of time tasks lose scheduling opportunities because a resource class is unavailable. It is the only metric that maps kernel-side waiting onto "how many tasks are affected", and its overhead is low enough for continuous collection.
/proc/pressure/io has two lines. some is the fraction of time at least one task was stalled on IO; full is the fraction of time every non-idle task was stalled on IO — meaning the box is fully serialized on disk. some reflects individual experience, full reflects systemic stall; both high is a catastrophe.
Combine PSI with cgroups. systemd services expose IOReadPressureIOPressure and friends via systemctl show <unit> -p IOReadPressureIOPressure, or you can read the cgroup io.pressure file directly. That answers "which service is dragging the machine down" rather than just "the machine is slow".
For monitoring, apply rate() to node_pressure_io_wait_seconds_total, normalize by time, and alert on thresholds (for example above 40% for 5 minutes). Track avg10/avg60/avg300 together to separate short spikes from sustained degradation.
cat /proc/pressure/io
cat /sys/fs/cgroup/system.slice/<unit>/io.pressure
systemctl show nginx.service -p IOReadPressureIOPressureAttributing to processes: who is actually reading and writing
iostat tells you the device is busy, not who is making it busy. Attribution needs pidstat -d 1, which reports kB_rd, kB_wr, kB_ccwr (cancelled writes — a high value means requests were issued then aborted, wasting bandwidth) and iodelay per process or thread.
pidstat output rolls: each refresh shows only the current interval, so watch for rows where kB_wr stays high rather than reading totals. Piping through sort does not work because output is streaming; in practice, watch a few screens and note PIDs that keep recurring.
fiotop is better for interactive triage: it lists processes ordered by IO rate with readable output, far more intuitive than pidstat. Its downside is that it redraws on a timer, so it is not scriptable and keeps no history.
When iotop is unavailable there is a dependency-free fallback: /proc/<pid>/io exposes per-process read_bytes / write_bytes counters (actual bytes to storage, excluding page-cache hits). Sample twice and difference them to get a real rate — handy inside minimal containers with nothing installed.
pidstat -d 1
# 关注 kB_wr 高、kB_ccwr 也高的行：写风暴 + 请求被取消
cat /proc/$(pgrep -n nginx)/io | grep -E "read_bytes|write_bytes"
fiotop -o -d 1 # -o 按写入速率排序Classify the root cause, then act
Small-file storm (high %util, low await, high %arq-sz): the classic inode and fsync-heavy workload where metadata operations saturate the queue. Fix by merging small files into a few large ones, or by switching to a storage format that supports direct IO. More RAM only defers the IO, it does not remove it.
Large sequential throughput (low await, high %arq-sz): almost always backup, sync, or export jobs. Verify whether the work is necessary, move it to a quiet window, or throttle it with ionice -c3 (idle priority) or cgroup IO weights. This class has the most tuning headroom and the largest quick win.
Heavy random writes (high await with a long latency tail): usually a database checkpoint or log flush. Check whether log/backup configuration exceeds realistic performance expectations, and whether a volume was mistakenly configured for synchronous commits. This is especially visible on network storage — review the sync mount options in /etc/fstab together with the storage-side cache policy.
Do not forget the kernel layer: confirm the scheduler at /sys/block/*/queue/scheduler (none suits high-concurrency NVMe, bfq suits desktop and low-concurrency workloads), tune depth at /sys/block/*/queue/nr_requests, and identify rotational disks at /sys/block/*/queue/rotational. Persist any change via /etc/sysctl.d/ or udev rules so the change survives reboot.
cat /sys/block/nvme0n1/queue/scheduler
# 高并发 NVMe 常用 none；低并发/桌面可试 bfq
cat /sys/block/nvme0n1/queue/nr_requests
cat /sys/block/nvme0n1/queue/rotational # 0 = 非机械盘