巧匠Developer Cheatsheet
SysOps

Linux CPU at 100%: the Shortest Path from top to the Thread Stack

"Restart fixes everything" is the most expensive option when CPU is pegged. This guide gives a reusable path: classify the load, pin the thread, capture what it is executing, then act — with dedicated tools for Java, Python and Go.

By 巧匠 Team·8 min read·Updated 2026-08-30

Step 1: user, system, or iowait

Read the CPU line at the top of `top` first: high us means application code is burning CPU (business logic, hot loops, GC); high sy means kernel overhead (syscall storms, lock contention, context switching); high wa is not CPU business at all — slow disk IO inflates load. The directions are totally different, and the wrong tool wastes an afternoon.

Compare load average against core count: `nproc` prints cores; only load persistently above that means real queuing.

top        # 看 %Cpu(s) 行的 us / sy / wa
nproc      # 核数，用于对照 load average

Step 2: pin the process, then drill to threads

After `top` (press P to sort by CPU) identifies the hot process, most people stop there. But the hotspot is often just a few threads inside one process — `top -H -p <pid>` shows a per-thread view; note the thread IDs burning CPU.

An even handier form is `pidstat -t -p <pid> 1`, which prints per-thread CPU usage once per second without an interactive terminal.

top -H -p <pid>          # 该进程的线程级视图
pidstat -t -p <pid> 1    # 逐秒采样各线程 CPU

Step 3: capture what that thread is executing

Java: convert the thread ID to hexadecimal and search `jstack <pid>` output for nid=0x<hex>; the stack frames appear directly. Capture two or three times — frames stuck at the top are the hotspot. Python: `py-spy dump --pid` gives the call stack without code changes or restarts. Go: grab a CPU profile if pprof is wired in. Universal fallback: `perf top -p <pid>` for kernel-side hot symbols.

printf '%x\n' <tid>          # 线程号转十六进制
jstack <pid> | grep -A 20 nid=0x<hex>
py-spy dump --pid <pid>     # Python 进程调用栈
perf top -p <pid>           # 通用内核态热点

Match against the usual root causes

① Infinite loops or a while whose exit condition never holds; ② catastrophic regex backtracking (nested quantifiers over long text); ③ frequent Young/Full GC (memory starvation masquerading as a CPU problem); ④ crypto, compression or serialization doing heavy work on the hot path; ⑤ synchronous log flushing at high QPS. Whatever the stack shows usually maps to one of these.

jstat -gcutil <pid> 1000   # 每秒看 GC 频率，判断是否 GC 风暴

Stop the bleeding first, cure later

Bleeding-stop actions for a live incident: drain traffic (remove the node from the load balancer), throttle, or kill the runaway process so the supervisor restarts it. Before killing, capture all evidence — thread stacks, GC logs — otherwise the clues vanish with the restart and the issue becomes "spooky and random".

The cure goes back to code: fix loop conditions, rewrite regexes, add caching, move heavy work off the hot path, size the heap or fix the leak. If the root cause is traffic growth, it is a capacity problem — scaling out is more honest than patching code.

jstack <pid> > /tmp/stack-$(date +%s).txt   # 先留证据
kill <pid>                                   # 再重启（supervisor 会拉起）

One-page recap: the chain

top to classify us/sy/wa → top -H / pidstat -t to pin the thread → jstack / py-spy / perf to capture the stack → match the five root causes → preserve evidence then stop the bleeding → cure in code. Drill this chain and average CPU triage drops under ten minutes.

top → top -H -p <pid> → jstack/py-spy → 定位 → 止血 → 根治

More guides

Git Accident Recovery: Fix Wrong Commits, Wrong Branches and Deleted BranchesThe scariest moment in daily work is "oh no, I messed up a commit or branch". This guide gives copy-ready recovery commands for real accident scenarios, with clear safety boundaries for each.Docker Image Slimming: 5 Steps from 1.2GB to 120MBBigger images mean slower pulls, larger attack surface and higher storage cost. This guide gives 5 immediately applicable slimming steps, each with a reusable Dockerfile snippet and a verification command.Nginx 502 Bad Gateway? Locate It in 5 StepsA 502 means Nginx itself is fine but got an invalid response from the upstream (backend/proxy). This guide gives the shortest path: process → config → timeout → resources → logs.Linux Disk Full: Locate the Real Culprit with df, du and lsofA "No space left on device" error does not mean you should delete files at random. This guide walks through the full path — filesystem water level, directory drill-down, and "deleted but still held" files — plus a safe cleanup order that will not destroy data.MySQL Slow Query Tuning: Slow Log, EXPLAIN and Indexing"The database is slow" usually means a handful of slow SQL statements. This guide offers a repeatable flow: use the slow log to circle the most expensive queries, read the plan with EXPLAIN, decide whether an index pays off, and spot the most common index-defeating patterns.Redis Slowdown Playbook: Big Keys, Blocking Commands and EvictionRedis processes commands on a single thread, so one blocking hotspot slows the whole instance. This guide follows the order "memory level → big keys → blocking commands → eviction and persistence", each step with copy-ready commands.Kubernetes CrashLoopBackOff: Diagnose a Restart-Looping Pod in 5 StepsCrashLoopBackOff is not one error but a state: the container dies on start, and Kubernetes retries with backoff. This guide gives the shortest path — exit code → previous logs → probes → resources → dependencies.SSH Passwordless Login: from ssh-keygen to Permission deniedPasswordless SSH boils down to one sentence: put the public key into the remote authorized_keys. What blocks you is usually the three hidden gates — file permissions, the home directory, and SELinux. This guide covers the full setup and a step-by-step debugging path.Python Virtual Environments in Practice: venv, Dependency Pinning and Cross-Platform Gotchas"Works on my machine" usually means mixed environments. This guide builds a clean flow with the stdlib venv: create, activate, pin and reproduce — plus the three frequent pitfalls: Windows activation, version drift and cross-platform dependencies.VSCode Remote Development: Remote-SSH Setup and Connection DebuggingA weak laptop, code on an intranet server, GPUs in the cluster — Remote-SSH keeps the UI local while code and terminal run remotely, feeling native. This guide covers setup, passwordless auth, common connection failures and experience tuning.
← Back to Guides