Linux Disk Full: Locate the Real Culprit with df, du and lsof

Step 1: is it blocks, or is it inodes

Start with df -h to check the water level of each mount. Focus on the line at 100% Use% and which directory it is mounted on — different mounts are separate filesystems, so deleting files on the wrong one achieves nothing.

If capacity looks fine yet you still get "No space left on device", inodes are probably exhausted: a huge number of tiny files (caches, sessions, mail queues) consumed all inodes. Verify with df -i and its IUse% column.

df -h          # Capacity usage (find mount points at 100%)
df -i          # inode usage (IUse% 100% = too many small files)

Step 2: drill down with du, do not scan from root

The right way to find big directories is "partition first, drill down later": inside the full mount, run du -xh --max-depth=1 to see one level only, compare subdirectory sizes, then step into the largest one. Two or three rounds usually pin down the culprit. The -x flag matters — it stays on one filesystem and avoids cross-mount pollution.

For a directory you found but cannot interpret, run du -sh <dir>/* | sort -rh | head to list the top ten by size; you will usually recognize whether it is logs, backups, or an app data directory.

du -xh --max-depth=1 /var 2>/dev/null | sort -rh | head
# Keep drilling into the largest subdirectory; 2-3 rounds is usually enough to pin it down

Step 3: df and du disagree? Find deleted-but-held files

A classic trap: df says 100% but du finds no large files. The cause is a process still holding a deleted file — it is gone from the directory, yet its data still occupies disk until the process releases it or restarts.

Use lsof +L1 to list files with link count zero that are still open, filtering by user or process name. Restarting the matching service (or a graceful reload of the logging process) releases the space immediately, without deleting any data.

lsof +L1                      # Files that were deleted but are still held open
lsof +L1 -u appuser            # Only show processes of one user
# Once confirmed: systemctl restart <service> to release the handles

Four usual suspects, ranked

(1) Application logs grow fastest and can be emptied with truncate -s 0 without interrupting the writing process (safer than rm, which creates the deleted-handle problem); (2) Docker: docker system df shows image/container/volume usage, docker system prune clears dangling resources; (3) Core dumps and temp dirs: /tmp, /var/tmp, core.*; (4) old backups: look for tarballs named by date.

truncate -s 0 /var/log/app/app.log   # Empty the file without interrupting the writing process
docker system df                     # Check image/volume usage
docker system prune                  # Remove dangling images and stopped containers

A safe cleanup order

Recommended order: truncate growing logs first (instant, zero data risk), then temp files, then docker prune, and only then consider deleting backups — before removing a backup, confirm the data can be rebuilt from the source. Never manually delete files inside a database data directory.

When the disk is full even root login can misbehave (shell history and temp files fail to write), so the emergency order is: free a small chunk of space first, then investigate calmly — do not spend ten minutes hunting the root cause first.

truncate -s 0 /var/log/messages   # The first emergency cut
journalctl --vacuum-size=200M    # Shrink systemd logs (a common big consumer)

Prevention: make fullness a non-event

Three things: (1) configure logrotate to rotate and compress app logs; (2) add monitoring alerts on disk usage (warning at 80%, critical at 90%) so it pages you before users notice; (3) for services that generate many small files fast, use a dedicated mount or scheduled cleanup so inode exhaustion cannot take down the root filesystem.

/etc/logrotate.d/app example:
/var/log/app/*.log {
  daily
  rotate 14
  compress
  missingok
  copytruncate
}