Linux Troubleshooting Cheatsheet - System Fault Diagnosis Command Reference
When a server misbehaves, the reliable approach is not recalling the right command but following a fixed descent order: processes, ports, disk, memory, logs. Processes and load start from uptime — remembering that on Linux this counts runnable plus uninterruptible processes, not CPU percent — then top or htop sorted by CPU and memory, and finally ps for the concrete PID. Ports and networking cover ss and netstat listeners, nc reachability probes, and using dig plus curl to distinguish a closed port from a broken service above it. Disk and memory use df -h, du -sh to walk down to the offending directory, and iostat with free -m to tell IO pressure from RAM pressure. Logs combine journalctl -u, systemctl status and tail -f to separate a service that is down from one failing repeatedly. Several classic misjudgements are flagged: a full disk with no large file usually means a process still holds a deleted inode, found with lsof +L1, and plain kill sends SIGTERM with -9 reserved for wedged processes.
Process & Load 5
top -cps aux --sort=-%cpu | headps aux --sort=-%mem | headkill -9 <pid>uptimePort & Network 5
ss -tlnpss -tlnp | grep :8080curl -v http://127.0.0.1:8080ping -c 4 hostdig +short example.comDisk & Memory 5
df -hdu -sh * | sort -rh | headfree -hlsof +L1iostat -x 1Logs & Services 5
journalctl -u nginx -n 200 --no-pagerjournalctl -fsystemctl status nginxtail -f /var/log/app.logdmesg -T | tailCommand Examples
Start with load: what the three numbers in uptime each cover
uptimeThe three figures are the 1, 5 and 15 minute averages, and they count tasks rather than a percentage. A 1-minute value far above the 15-minute one means the pressure just arrived; all three high means sustained saturation.
Output
14:32:11 up 12 days, 3:07, 2 users, load average: 1.52, 1.18, 0.94
Two shapes of a full disk: df for totals, du for the heavy hitters
df -h
du -sh * | sort -rh | headdf is the filesystem view and only shows totals; du walks directories and can rank the heavy ones. If the two disagree by a wide margin, the space is held by deleted files that are still open — switch to lsof +L1 at that point.
Output
Filesystem Size Used Avail Use% Mounted on /dev/vda1 40G 39G 1.0G 98% / /dev/vdb1 200G 148G 52G 74% /data 24G ./build 11G ./node_modules 3.1G ./logs
Confirm who owns the port: filter with ss, then cross-check the PID
ss -tlnp | grep :8080No output means nothing is bound. When there is output, read the process name and PID in the users section, then confirm with top -c or ps that this PID really is the expected service rather than a stray debug process.
Output
LISTEN 0 511 0.0.0.0:8080 0.0.0.0:* users:(("node",pid=20481,fd=20))Memory or IO: free for headroom, dmesg for evidence of an OOM kill
free -h
dmesg -T | tailIt is the available column that matters; a small free figure alone does not mean memory pressure. An OOM line in dmesg means the process was killed by the kernel, so look at memory limits and leaks. If dmesg is clean but load is still high, redirect the investigation to disk saturation with iostat -x 1.
Output
total used free shared buff/cache available Mem: 7.7Gi 6.1Gi 321Mi 12Mi 1.3Gi 1.1Gi Swap: 8.0Gi 2.4Gi 5.6Gi [Sat Oct 4 02:11:07 2026] Out of memory: Killed process 20481 (node) total-vm:8123456kB, anon-rss:5201440kB
Common Pitfalls
- kill -9 sends SIGKILL, which gives the process no chance to release file locks, flush buffers or close connections; stateful services restarted after a hard kill lose whatever was not written to disk. Try a plain kill first and allow a graceful shutdown window.
- A high load average does not mean the CPU is insufficient: uninterruptible IO wait counts toward load too. Rule out disk saturation with iostat -x 1 before deciding to add capacity.
- lsof +L1 walks every open file descriptor on the machine, which is expensive on a process-heavy host. Run it on demand during an incident and interrupt it rather than leaving it polling.
- journalctl -f and tail -f hold the terminal indefinitely. Exit with Ctrl-C when the investigation is over, otherwise a quiet log looks exactly like a hung command.
- df reports the filesystem view shared by all users, so when the root filesystem fills up system logging can fail at the same time. When a service restarts for no reason, check the disk and the logs together.
Tips
- Disk shows full but no large file found — a process likely holds a deleted file. Use lsof +L1 to locate it, then restart the process.
- kill sends SIGTERM by default for graceful shutdown — try regular kill first, only use kill -9 if stuck.
- "Out of memory: Killed process" in dmesg means OOM, not a crash in the application itself.
FAQ
uptime reports a high load average while top shows an idle CPU. Is that contradictory?
No. On Linux the load average counts runnable tasks plus tasks in uninterruptible sleep, which is where disk-blocked work usually parks. It is not a CPU percentage. A process stuck waiting on IO therefore keeps the load figure high while the CPU sits idle waiting for it. So high-load-with-idle-CPU points at storage latency first, not at insufficient compute — confirm it with iostat -x 1 and look at queue depth and %util.
How do I confirm whether a port is actually being listened on, and by whom?
Two steps. First run ss -tlnp | grep :8080: -l restricts to listening sockets, -t to TCP, -n skips name resolution, and -p adds the owning process. No output at all means nothing is bound to that port, which immediately rules out a server-side problem. If you do get a line, read the process name and PID in the -p column and check they belong to the service you expected.
The port is listening but curl to the local address fails. What do I check next?
A port problem is now off the table, so look at the handshake. curl -v http://127.0.0.1:8080 prints the full request and response exchange: whether it stalls during the TLS handshake, whether the connection is reset right after the request line goes out, and what status code comes back. Those three shapes point at different causes — a stuck handshake suggests protocol or certificate trouble, an immediate reset usually means the process aborted internally or has an unhandled connection hook, and a real response with a bad status is an application-level issue.
df says the partition is full but du cannot find any large file. What is going on?
This is the classic case of space held by deleted-but-open files. When a process holds a file open and the file is unlinked, the blocks are only released once the process drops the descriptor, so the filesystem looks full while du — which walks live directory entries — sees nothing. lsof +L1 is the right tool: it lists open files whose link count is zero or one. Restart the owning process (plain kill first, kill -9 only if it refuses to exit) and the space returns immediately; for a long-running service, also work out who keeps writing files and never cleaning up.
A service vanished with no explanation. What should I look for in dmesg?
Look for a kernel-side termination record. dmesg -T | tail dumps the tail of the kernel ring buffer in human-readable timestamps. A line such as Out of memory: Killed process means the kernel OOM killer took the process down rather than the application dying on its own, and the fingerprint is distinctive: the victim is the largest memory consumer and the same buffer shows allocation failures. That shifts the investigation from application logs to memory limits or a leak.
Official References
Each command links to its official documentation below, so you can verify the latest usage and read deeper.
Maintained by LaoHand
Publicly updated on Jul 21, 2026, continuously proofread against official docs.
Contact Us
Wrong command or description? Send us corrections, business inquiries or product feedback by email.
Contact Us