Two independent ledgers: data blocks and inodes
Every POSIX-permission Linux filesystem maintains two resource classes: **data blocks**, which store file contents, and **inodes**, which store metadata — permissions, ownership, timestamps, size, and pointers to data blocks. The inodes themselves live on disk but their total count is **fixed at format time**.
Because the inode total is fixed, every inode can be consumed while not a single data block is used. That is the entire explanation behind "No space left on device at 20% usage": the capacity ledger is fine, the file-count ledger is exhausted.
A further trap: inode exhaustion reports the **exact same** errno as space exhaustion (ENOSPC). The kernel does not distinguish the cause, so the only reliable habit is to always read `df -h` and `df -i` together. Looking at one alone will mislead you.
Symlinks, hard links, empty files, and directories each consume one inode, including zero-byte files. That is why a 0-byte file can be the culprit: it occupies no data blocks yet still burns an inode. Enough of them will exhaust an entire filesystem.
df -h # 数据块水位
df -i # inode 水位，IUse% 达到 100% 即耗尽
# 两条一起看，缺一不可Which workloads are most dangerous
① Massive small-file counts: static asset directories, build artifacts (`node_modules`, `.venv`, `target/`), backup extraction dirs. Each file burns an inode while using very few blocks — the archetypal "capacity empty, inodes full". With a 4 KB block and 1 KB of content, three quarters of the block is wasted per inode.
② Session and cache directories: PHP/Java session dirs, Redis persistence shards, application local caches (`~/.cache`), Nginx `proxy_cache`. Short file lifetimes plus fast growth mean a single batch job can write hundreds of thousands of tiny files overnight.
③ Mail and queue spools: `/var/spool/postfix`, mail queues, unarchived log segments. One file per message, several per message with attachments — steep growth under bursts.
④ Container overlay layers: every image layer and every container writable layer counts against the overlay inode budget, so a laptop running dozens of local images can hit this easily. It is easy to miss because containers are intuitively assumed to live elsewhere.
df -i /
# 典型输出：iused 100%，avail 为 0，而 df -h 仍有大量剩余容量Locating it: counting files layer by layer
The approach mirrors big-directory hunting, replacing `du` with `find | wc -l`. Keep the "partition first, drill down later" strategy — a full-disk find on an inode-starved machine can take tens of minutes of IO.
Start with a one-level count inside the full mount. Expect a small discrepancy against the used count from `df -i` because directories themselves consume inodes; do not be confused when the numbers do not line up exactly.
Step into the largest directory and repeat; two or three rounds usually pin it down. The fastest signal is a large ratio of "file count versus apparent data size": a directory with 500,000 files occupying 200 MB must be dominated by its inode usage.
Often you do not need precise location — running `ls | wc -l` on a known suspect (session dir, cache dir, `node_modules`) confirms it immediately. Before doing this on production, make sure `find` will not trigger heavy directory reads; add `-xdev` to stay within the mount.
find /var -xdev -type f -o -type d 2>/dev/null | wc -l
# 逐层下钻，对比每个子目录的文件数
for d in /var/*; do printf "%s " "$(find "$d" -xdev 2>/dev/null | wc -l)"; echo "$d"; done | sort -rn | headCleaning up safely and expanding
The first rule of cleanup is "delete regenerable small files, never touch data". Caches, sessions, temp files, and build artifacts are regenerable: removing them lets the app rebuild at the cost of a brief slowdown. Database files, user uploads, and archived logs are not — even if they are small files, deleting them is a data incident.
Prefer the application cleanup command over `rm -rf`. Redis has `SCAN` for incremental non-blocking deletion, Nginx has `proxy_cache_purge`, and package managers have `apt clean` / `yum clean all`. A raw `rm -rf` over hundreds of thousands of files generates heavy directory operations that can hang or fail outright on an already inode-starved filesystem.
Confirm up front that cleanup actually frees resources: inodes are reclaimed immediately on unlink, with no restart or sync required — the same as disk blocks. So you can clean incrementally and watch `df -i` as you go.
If cleanup cannot free enough, or the file count is genuinely required by the business, the only real options are expansion or a different filesystem. ext4 fixes its inode count at format time and cannot change it online; XFS can grow online with `xfs_growfs` but its inode ceiling is likewise fixed at format time. Note that filesystems like `tmpfs` take their size from the mount option, so remounting with a larger `size=` is a fast mitigation.
du -sh /var/cache/* 2>/dev/null | sort -rh | head
apt clean
rm -rf /var/lib/app/sessions/* # 可再生内容，可安全清理
df -i / # 清理后立即复查
mount -o remount,size=4G /dev/sdb1 # tmpfs 类可直接放大Prevention: monitoring and limits
Make inode usage a first-class alert item rather than folding it into disk usage. Typical thresholds are 80% warning and 90% critical. Note that inode consumption is **step-wise**: once you are alerted, you are often close to 100%, which leaves far less reaction time than capacity exhaustion does.
Give small-file-heavy directories their own quota or mount point. Putting sessions, caches, and log spools on a separate ext4/xfs mount means exhausting them cannot take down the root partition. systemd also offers `TemporaryFileSystem=/var/cache/app`, which keeps such files in an in-memory filesystem that vanishes when the process exits.
The application owns part of the fix: change the "one file per record" pattern into batched files (merged log segments, a database instead of a file tree) or enforce a minimum file size — many exports can simply ship as a tarball. Such changes remove inode pressure structurally rather than marginally.
Finally, discipline: after every "No space left on device" incident, leave behind a one-line conclusion — capacity or inodes? The next engineer on call should be able to answer in ten seconds instead of spending an hour re-deriving it.
# 告警表达式思路：inode 使用率独立成一条规则
# node_filesystem_files_free / node_filesystem_files < 0.1 即已耗尽
systemctl show app.service -p TemporaryFileSystem