巧匠Developer Cheatsheet
SysOps

Linux Network Troubleshooting Field Manual: Connectivity, Packet Loss, Latency, and DNS

From unreachable pings to crawling slowness and flaky domain resolution, this guide walks layer by layer—link, IP, transport, application—showing which command and which field to look at, with copy-run experiments along the way.

By LaoHand Team·8 min read·Updated 2026-09-06

Draw a probe path first: every hop from local host to peer

Troubleshooting fails most often when there is no layering. Facing "cannot reach the Internet", first decide whether it is a single node or the whole path: whether the interface has an IP (ip a), whether a default route exists (ip route), whether the gateway answers ping, then climb upward one layer at a time. Use the layer-appropriate tool and never let curl guess about link loss.

Here is the standard command sequence from link to application, each step with a clear pass criterion: interface UP with an address, a route exists, gateway reachable, DNS resolves, TCP three-way handshake succeeds. Whichever step fails scopes the problem to that layer.

ip a                       # 网卡地址/UP 状态
ip route                   # 默认路由
ping -c 3 192.168.1.1      # 网关
ping -c 3 223.5.5.5        # 公网 IP（阿里 DNS）
dig +short example.com     # 域名解析
curl -vI https://example.com --connect-timeout 5

Loss and latency: ping both ends, read the stats, and look past ICMP

ping only measures ICMP, and many operators throttle or drop ICMP, so a little loss in ping is normal. What really matters is the TCP business traffic. When reading ping, watch mdev (jitter), duplicate packets, and the TX side TXerrors/TXdropped—loss happening at your egress versus at the peer requires very different treatments.

To pin down a specific path, use mtr hop-by-hop to watch loss and avg, turning an abstract "network is slow" into a concrete bad router in the middle. Then at TCP layer use ss -i for retransmission rates.

ping -c 100 -i 0.5 1.1.1.1 | tail -n 3
mtr -rwbc 50 1.1.1.1
# TCP 重传/丢包视角
ss -it | grep -E "retrans|rto|rtt"
# 网卡出口统计
ethtool -S eth0 | grep -iE "tx_error|tx_dropped|collisions"

Stuck at transport: SYN dropped vs handshake timeout, triage with nc/ss

When ping works but a service does not connect, the problem is usually at the port layer. First confirm with ss -ltn whether the port is listening and whether it binds to 0.0.0.0 or only loopback; a surprising share of "port not reachable" cases are just the service bound to 127.0.0.1. Then bypass the firewall with tcping or nc for a raw TCP probe.

Dropped SYN at the half-open state usually points to firewalls or syncookies; connections that establish but reset frequently point to a full backlog (listen queue overflow), mirrored by a piling Recv-Q in ss -ltn and the kernel setting tcp_max_syn_backlog. Below is a round of triage.

ss -ltn | grep -E ":80 |:443 |:3306 "
ss -lnt | awk "! /127.0.0.1/ {print}"  # 排除只回环的服务
nc -vz -w 3 10.0.0.8 3306
# 看 listen 队列是否溢出
ss -lt 'sport = :80' | grep -c SYN-RECV
cat /proc/sys/net/ipv4/tcp_max_syn_backlog

DNS: three ways to read slow, failing, or poisoned resolution

Domain problems usually show as "browser occasionally fails to open". Read resolution latency straight from dig: a big gap between two query time values means the upstream resolver is flaky. Distinguish failures by code—NXDOMAIN means the name truly does not exist, SERVFAIL means upstream trouble or insufficient authority.

A huge chunk of DNS pain is the search domain suffix: with option ndots in effect, `curl good.com` may append a search suffix and trigger extra lookups, so what looks like slowness is actually doubled query count. Use +trace to walk the whole recursion chain and +short to grab the A record fast.

dig +short example.com
dig +trace example.com A
dig example.com | grep -E "Query time|flags:"
nslookup example.com 223.5.5.5
cat /etc/resolv.conf
# 统计每类解析耗时
for d in 223.5.5.5 8.8.8.8 114.114.114.114; do echo "-- $d"; dig +time=2 +tries=1 @$d example.com | grep "Query time"; done

Mistake vs fix: do not dig a hole with iptables -F

When a server suddenly stops answering, the gut reaction is "flush the firewall". But iptables -F only clears rules in the default filter chains; if you rely on firewalld/nat or custom chains, it leaves a half-flushed state—INPUT emptied while FORWARD rules linger, or conversely your own traffic starts bleeding sideways.

The safest route is to know where rules come from: check systemctl status firewalld to see if firewalld owns them, or if you use ufw walk it with ufw status numbered. Back the chain rules to a file before touching anything, keep a second terminal open to confirm your session is not cut, then test the business port.

# 错误示范：无差别清空惹祸
# iptables -F  # 会清空 filter 现有规则

# 修复对照：先备份再精确操作
iptables-save > /tmp/fw.$(date +%F).bak
# 只对指定链/端口放开，不动其它
iptables -I INPUT -p tcp --dport 443 -j ACCEPT
# 改完立刻验证
ss -lnt | grep :443
curl -sI https://example.com -o /dev/null -w "%{http_code}\n"

Verify a full path: tcpdump a capture and read the handshake

Finish with tcpdump for a definitive answer: capture twenty packets and seeing the three-way handshake (SYN/SYN-ACK/ACK) proves the path and port are fine and the trouble is higher up; SYN with no reply means firewall or a non-listening peer; SYN→ACK but no follow-up data suggests a middlebox or MTU mismatch.

While capturing, watch for DF-marked packets—large segments silently dropped after fragmentation often show up as "small requests fine, bigger responses time out". After verifying, stop tcpdump promptly so it does not keep logging to the background forever.

sudo tcpdump -i eth0 -n host 10.0.0.8 and tcp port 443 -c 20
sudo tcpdump -i any -n 'tcp[tcpflags] & (tcp-syn) != 0 and host 10.0.0.8' -c 10
# 看 DF 分片
sudo tcpdump -i eth0 -n -e 'and (ip[6] & 0x40)' 2>/dev/null | head

Official References

Each command links to its official documentation below, so you can verify the latest usage and read deeper.

More guides

Git Accident Recovery: Fix Wrong Commits, Wrong Branches and Deleted BranchesThe scariest moment in daily work is "oh no, I messed up a commit or branch". This guide gives copy-ready recovery commands for real accident scenarios, with clear safety boundaries for each.Docker Image Slimming: 5 Steps from 1.2GB to 120MBBigger images mean slower pulls, larger attack surface and higher storage cost. This guide gives 5 immediately applicable slimming steps, each with a reusable Dockerfile snippet and a verification command.Nginx 502 Bad Gateway? Locate It in 5 StepsA 502 means Nginx itself is fine but got an invalid response from the upstream (backend/proxy). This guide gives the shortest path: process → config → timeout → resources → logs.Linux Disk Full: Locate the Real Culprit with df, du and lsofA "No space left on device" error does not mean you should delete files at random. This guide walks through the full path — filesystem water level, directory drill-down, and "deleted but still held" files — plus a safe cleanup order that will not destroy data.Linux CPU at 100%: the Shortest Path from top to the Thread Stack"Restart fixes everything" is the most expensive option when CPU is pegged. This guide gives a reusable path: classify the load, pin the thread, capture what it is executing, then act — with dedicated tools for Java, Python and Go.MySQL Slow Query Tuning: Slow Log, EXPLAIN and Indexing"The database is slow" usually means a handful of slow SQL statements. This guide offers a repeatable flow: use the slow log to circle the most expensive queries, read the plan with EXPLAIN, decide whether an index pays off, and spot the most common index-defeating patterns.Redis Slowdown Playbook: Big Keys, Blocking Commands and EvictionRedis processes commands on a single thread, so one blocking hotspot slows the whole instance. This guide follows the order "memory level → big keys → blocking commands → eviction and persistence", each step with copy-ready commands.Kubernetes CrashLoopBackOff: Diagnose a Restart-Looping Pod in 5 StepsCrashLoopBackOff is not one error but a state: the container dies on start, and Kubernetes retries with backoff. This guide gives the shortest path — exit code → previous logs → probes → resources → dependencies.SSH Passwordless Login: from ssh-keygen to Permission deniedPasswordless SSH boils down to one sentence: put the public key into the remote authorized_keys. What blocks you is usually the three hidden gates — file permissions, the home directory, and SELinux. This guide covers the full setup and a step-by-step debugging path.Python Virtual Environments in Practice: venv, Dependency Pinning and Cross-Platform Gotchas"Works on my machine" usually means mixed environments. This guide builds a clean flow with the stdlib venv: create, activate, pin and reproduce — plus the three frequent pitfalls: Windows activation, version drift and cross-platform dependencies.VSCode Remote Development: Remote-SSH Setup and Connection DebuggingA weak laptop, code on an intranet server, GPUs in the cluster — Remote-SSH keeps the UI local while code and terminal run remotely, feeling native. This guide covers setup, passwordless auth, common connection failures and experience tuning.Bash Script Safety: set -euo pipefail and trap for Trustworthy ScriptsA Bash script that "finishes" is not a script that "worked": failed commands do not stop it, empty variables pass silently, and pipes hide upstream crashes. This guide fixes all of them with a copy-ready skeleton (set -euo pipefail + trap + quoting discipline).YAML Gotchas: Type Coercion, Indentation and Multiline StringsYAML files that "look right but run wrong" usually fail in three places: silently coerced scalars, misplaced indentation, and ambiguous multiline strings. This guide dissects each trap with detection methods and a toolchain.HTTP Caching in Practice: Configuring Cache-Control and ETag CorrectlyMisconfigured caching ends one of two ways: users never see updates, or origin traffic explodes. This guide covers the split between freshness and validation caching, the hashed-asset vs HTML strategy divide, plus ready-to-use Nginx config and verification commands.Git Merge and Rebase Conflict Resolution in PracticeFor developers branching daily, this guide reads merge conflict markers end to end with reproducible cases and shows how to handle merge, rebase, and rename conflicts.Bisecting: Pinpoint the Commit That Introduced a BugFor anyone tired of hunting commits by gut feel, this is the binary-search way to find the commit that broke something, including making the check scriptable so bisect runs hands-free.Git Submodule Management and Its Common TrapsA field guide to when a submodule actually makes sense, how to add, clone, and update it, and the standard escape hatches for empty dirs after clone, branch drift, and nesting.Parallel Development with git worktree: Multiple Branches at OnceFor devs juggling several branches who fear stash chaos while switching, here is how to keep one clean worktree per task and never stash-swap again.docker compose Multi-Service Integration and Startup TroubleshootingGetting to the bottom of ordering, environment variables, port collisions, and service-name DNS inside the compose network, with copyable yaml and debugging commands.Kubernetes HPA: Configuring and Verifying AutoscalingFor engineers comfortable with Deployments but new to autoscaling: from installing metrics-server, writing an HPA, to load-testing and actually watching replicas scale out.Dockerfile Security Hardening: Shrinking the Attack SurfaceA landing checklist for image maintainers and service containerizers: non-root user, read-only root FS, dropping capabilities, layer hygiene, and image scanning.JavaScript Error Handling: From try/catch to Global Fallbacks and RetryA practical catalog for backend and full-stack JS: catching sync and async errors, Promise unhandledrejection, global fallbacks, and exponential-backoff retries.Python Packaging and Dependency Locking: From pyproject.toml to PyPIFor turning scripts into publishable packages: pyproject.toml for metadata, venv isolation, lock files for reproducible environments, and releasing to PyPI with twine.Linux Memory Exhaustion and OOM Hunting: From dmesg to the Process Likely Out of ControlWhen a server claims it is out of memory it rarely tells the whole truth; this guide shows how to use free, /proc/meminfo, cgroups, and dmesg to close in on the process actually eating memory, and how to configure against recurring OOM kills.Debugging systemd Service Startup Failures: From status and Dependencies to Ordering Deadlockssystemctl is-failed is just the first step; this guide splits failures into four kinds—launcher error, process instantly exiting, dependency not ready, and port never listening—with the right log entry and fix for each.Linux File Permissions and sudo in Practice: chmod/ACL/setuid and Privilege BoundariesFrom bit permissions to ACLs to why setuid exists, this guide clarifies exactly how read/write/execute are computed, when the special bits backfire, and why sudo should not be a blanket ALL—plus a safer habit before you break something.Nginx Reverse Proxy in Practice: location Priority, proxy_pass, and Upstream Balancing Done RightMisconfigured proxies usually come down to location matching order and trailing-slash traps; this guide corrects the most commonly botched configs line by line and walks you through wiring a real Node backend into a load-balanced reverse proxy.HTTPS Certificates and Let's Encrypt Auto-Renewal: From Issuance to a cron FallbackManually requesting a cert is easy; the hard part is renewing it before expiry automatically. This guide walks certbot through webroot and HTTP-01 validation, configures renewal on port 80, and pairs it with two fallbacks so missing a renewal still does not take the site down.Debugging REST APIs End-to-End with curl: Methods, Headers, Auth, Upload, and Diagnosiscurl is the first scene of most API debugging; this guide starts from "see exactly what a request sent", covering GET/POST, query vs body, Bearer/JWT auth, Multipart uploads, and reading status code plus timing from one response.MySQL Logical Backup and Restore: mysqldump, Faster Imports, and Recovering from Accidental DeletesExporting with mysqldump is easy; the hard parts are whether the dump restores cleanly, whether the restores fast enough, and whether an accidental delete comes back. This guide covers consistency flags, single-transaction isolation, import-acceleration knobs, and a binlog-based recovery flow.PostgreSQL Slow-Query Diagnosis and Tuning: From EXPLAIN to shared_buffersSlow query tuning is a three-step ladder: find the query, read the plan, tune the config. This guide covers enabling the slow log, telling Seq Scan from Index Scan in EXPLAIN, and the core knobs that give the best bang for buck on buffers and concurrency.Cache Penetration, Breakdown and Avalanche: Identify and Protect Three Redis FaultsWhen your API suddenly slows down or your database collapses, three kinds of cache faults are usually to blame. This guide uses realistic log symptoms to help you tell apart penetration, breakdown and avalanche, then lays out practical defenses including mutex locks, null-caching and never-expiring hot keys.MySQL Index Design and EXPLAIN Analysis: From Wasteful to Zero Slow QueriesYou keep adding indexes yet slow queries do not go away — that usually means the indexes themselves are misdesigned. Starting from real business SQL, this guide covers composite index column order, covering indexes and index-failure scenarios, then walks through EXPLAIN output section by section so you can actually read a plan.tmux Session Management and Reuse: Remote Dev That Survives SSH DropsOne dropped SSH and a half-run build, open files and a dozen panes are all gone. This guide shows how tmux named sessions, detach/attach reuse and window/pane layout let you reconnect after a network drop and pick up exactly where you left off.VSCode Debugging Primer: From launch.json to Hovering Over a VariableCompared with sprinkling console.log everywhere, debugging lets you pause and inspect the call stack and every variable at each step. Starting from editing launch.json, this guide gets Node and front-end projects debugging, then walks through the sidebar, watch expressions and conditional breakpoints.Environment Variables and .env Management: Practical Details Beyond 12-FactorSecrets in code, environment variables drifting across environments, and .env committed to git — almost every project trips on these. Following the 12-factor principle, this guide clarifies precedence, injection methods, and how to manage local and CI config safely.Prettier and ESLint Together: Formatting and Linting That Do Not FightPrettier handles formatting and ESLint enforces quality, but when their rules overlap they fight each other and pre-commit runs get messy. This guide explains how to divide responsibilities, turn off conflicting rules, unify config, and enforce everything in CI.Take Claude Code Through a Real Bug-Fix FlowAn AI coding assistant is not "let it auto-fix" but a disciplined workflow. Using a real project bug, this guide walks the full loop from reproduction, adding a test, surgical fixes, to regression verification, with the exact instructions to issue at each step.Prompt Engineering for Coding Agents: Make Claude Code / Codex a Reliable CollaboratorGetting a coding agent right on the first try is not about the model but the instruction quality. This guide explains context constraints, minimal-change rules, test-first practice and review prompts, so you can craft reproducible prompts with fewer hallucinations.
← Back to Guides