Step 1: is it blocks, or is it inodes
Start with `df -h` to check the water level of each mount. Focus on the line at 100% Use% and which directory it is mounted on — different mounts are separate filesystems, so deleting files on the wrong one achieves nothing.
If capacity looks fine yet you still get "No space left on device", inodes are probably exhausted: a huge number of tiny files (caches, sessions, mail queues) consumed all inodes. Verify with `df -i` and its IUse% column.
df -h # 容量水位（找 100% 的挂载点）
df -i # inode 水位（IUse% 100% = 小文件太多）Step 2: drill down with du, do not scan from root
The right way to find big directories is "partition first, drill down later": inside the full mount, run `du -xh --max-depth=1` to see one level only, compare subdirectory sizes, then step into the largest one. Two or three rounds usually pin down the culprit. The `-x` flag matters — it stays on one filesystem and avoids cross-mount pollution.
For a directory you found but cannot interpret, run `du -sh <dir>/* | sort -rh | head` to list the top ten by size; you will usually recognize whether it is logs, backups, or an app data directory.
du -xh --max-depth=1 /var 2>/dev/null | sort -rh | head
# 找到最大子目录后继续下钻，2~3 轮即可锁定Step 3: df and du disagree? Find deleted-but-held files
A classic trap: `df` says 100% but `du` finds no large files. The cause is a process still holding a deleted file — it is gone from the directory, yet its data still occupies disk until the process releases it or restarts.
Use `lsof +L1` to list files with link count zero that are still open, filtering by user or process name. Restarting the matching service (or a graceful reload of the logging process) releases the space immediately, without deleting any data.
lsof +L1 # 已删除但仍被占用的文件
lsof +L1 -u appuser # 只看某用户的进程
# 确认后：systemctl restart <service> 释放句柄Four usual suspects, ranked
① Application logs grow fastest and can be emptied with `truncate -s 0` without interrupting the writing process (safer than rm, which creates the deleted-handle problem); ② Docker: `docker system df` shows image/container/volume usage, `docker system prune` clears dangling resources; ③ Core dumps and temp dirs: `/tmp`, `/var/tmp`, `core.*`; ④ old backups: look for tarballs named by date.
truncate -s 0 /var/log/app/app.log # 清空但不打断写入进程
docker system df # 查看镜像/卷占用
docker system prune # 清理悬空镜像与停止容器A safe cleanup order
Recommended order: truncate growing logs first (instant, zero data risk), then temp files, then docker prune, and only then consider deleting backups — before removing a backup, confirm the data can be rebuilt from the source. Never manually delete files inside a database data directory.
When the disk is full even root login can misbehave (shell history and temp files fail to write), so the emergency order is: free a small chunk of space first, then investigate calmly — do not spend ten minutes hunting the root cause first.
truncate -s 0 /var/log/messages # 应急第一刀
journalctl --vacuum-size=200M # systemd 日志瘦身（常见大头）Prevention: make fullness a non-event
Three things: ① configure logrotate to rotate and compress app logs; ② add monitoring alerts on disk usage (warning at 80%, critical at 90%) so it pages you before users notice; ③ for services that generate many small files fast, use a dedicated mount or scheduled cleanup so inode exhaustion cannot take down the root filesystem.
/etc/logrotate.d/app 示例：
/var/log/app/*.log {
daily
rotate 14
compress
missingok
copytruncate
}