巧匠Developer Cheatsheet
SysOps

Debugging systemd Service Startup Failures: From status and Dependencies to Ordering Deadlocks

systemctl is-failed is just the first step; this guide splits failures into four kinds—launcher error, process instantly exiting, dependency not ready, and port never listening—with the right log entry and fix for each.

By LaoHand Team·7 min read·Updated 2026-09-06

Get first-hand status: status is not the log

The beginner's habit is pasting systemctl status output into a forum, when status only gives a snapshot: Active state, main PID, and the last few exit codes. The few log lines it embeds are just "recent" and are easily overwritten by a later successful restart.

What you actually want is the full journal: journalctl -u servicename without a tight -n cap, sinking to the first error the service printed at startup—an assertion or a bad path usually appears in the first few lines. Also compare behavior between boot-time first start and a manual systemctl restart, because boot time often fails on ordering.

systemctl status myapp.service
systemctl is-failed myapp.service
journalctl -u myapp.service -n 500 --no-pager
# 只看启动那条线的完整输出
journalctl -u myapp.service --since=-2h | grep -iE "error|fail|denied|cannot"

Key points for diagnostic(Assistant): four failure kinds

Kind one, "launcher error", is when systemd cannot exec at all: often a bad bin path, wrong permissions, or a mistyped shebang, and the journal shows Failed to execute. Kind two, "instant exit", means the main process started up and crashed within seconds—the crash stack lives in the app log, while status shows exit-code and uptime.

Kind three, "dependency not ready", typically bites when your service starts but the Redis/Mongo it should connect to is still booting, so several services appear to wait on each other. Kind four, "port never listening", is the classic bind failure—address in use or insufficient privilege, and ss -lnt never shows the entry. With these four pinned down, "why it won't start" collapses to one line.

ls -l /usr/lib/systemd/system/myapp.service
# 一二类的证据栏
cat /etc/systemd/system/myapp.service | grep -E "ExecStart|ExecStartPre"
ss -lnt | grep 8080
# 三类的依赖链
systemctl list-dependencies myapp.service

Dependency and ordering deadlocks: common misuses of After and Requires

Many treat After as "wait until I am up" and Requires as "I depend on it", but that is upside down: Requires only says "without it I refuse to start", while After expresses ordering. Requests with only Requires and no After let two services race to start in parallel, and the weak dependency silently fails.

The nastier failure is the ordering deadlock: if A has After=B and B has After=A, systemd rejects the whole transaction and drops both units together. When the system seems stuck in boot-pending, run systemd-analyze verify to spot cycles and systemd-analyze blame to see which unit drags the boot.

# 错误示范：只有 Requires 没有 After 导致弱依赖竞速
[Unit]
Requires=redis.service

# 修复对照：补上顺序与依赖语义
[Unit]
Requires=redis.service
After=redis.service network-online.target
Wants=network-online.target

# 校验周期/耗时
systemd-analyze verify myapp.service
systemd-analyze blame | head -10

The sandbox options fault: ProtectedPaths/User locking the service in

systemd's sandbox options (ProtectSystem, ProtectHome, ReadOnlyPaths, User=) are great, yet they are also the sneaky cause of "looks configured but will not start". ProtectSystem=strict forces every path writable in your config under /var into read-only, and a database fails at the very first CREATE TABLE.

The fix is to halve first, then peer: comment out the suspicious isolation lines while keeping User/Group and core paths, confirm startup, then relax stepwise. Use systemd-analyze security to see the unit attack-surface score, and remember that a + prefix can explicitly allow a specific path without loosening the global rule.

systemd-analyze security myapp.service
# 错误示范：把 /var/lib 误设为只读
# ProtectSystem=strict

# 修复对照：只读保护 + 显式放行业务目录
ProtectSystem=strict
ReadWritePaths=/var/lib/myapp /run/myapp
# 或在路径前用 + 允许
[Service]
User=myapp
ProtectHome=read-only

Verification: checklist from failed state to sustained running

After fixing, do not stop at a green Active: active (running). Confirm three things: a reboot shows active (running) with no escalating restart counter; the port stays listening in ss for at least 60 seconds; and the journal shows no new errors. For dependency-style faults, stop the dependency once and start it again to see whether the ordering reproduces deterministically.

If the service still exits occasionally, run systemctl reset-failed to drop stale counters before observing again, so historical failures are not mistaken for new ones. Finally, archive the before/after startup times as a regression baseline.

systemctl reset-failed myapp.service
systemctl restart myapp.service
# 观察 60s
sleep 60
systemctl show myapp.service -p NRestarts -p ActiveState
ss -lnt | grep 8080
journalctl -u myapp.service -n 30 --since=-5min | grep -c error

Official References

Each command links to its official documentation below, so you can verify the latest usage and read deeper.

More guides

Git Accident Recovery: Fix Wrong Commits, Wrong Branches and Deleted BranchesThe scariest moment in daily work is "oh no, I messed up a commit or branch". This guide gives copy-ready recovery commands for real accident scenarios, with clear safety boundaries for each.Docker Image Slimming: 5 Steps from 1.2GB to 120MBBigger images mean slower pulls, larger attack surface and higher storage cost. This guide gives 5 immediately applicable slimming steps, each with a reusable Dockerfile snippet and a verification command.Nginx 502 Bad Gateway? Locate It in 5 StepsA 502 means Nginx itself is fine but got an invalid response from the upstream (backend/proxy). This guide gives the shortest path: process → config → timeout → resources → logs.Linux Disk Full: Locate the Real Culprit with df, du and lsofA "No space left on device" error does not mean you should delete files at random. This guide walks through the full path — filesystem water level, directory drill-down, and "deleted but still held" files — plus a safe cleanup order that will not destroy data.Linux CPU at 100%: the Shortest Path from top to the Thread Stack"Restart fixes everything" is the most expensive option when CPU is pegged. This guide gives a reusable path: classify the load, pin the thread, capture what it is executing, then act — with dedicated tools for Java, Python and Go.MySQL Slow Query Tuning: Slow Log, EXPLAIN and Indexing"The database is slow" usually means a handful of slow SQL statements. This guide offers a repeatable flow: use the slow log to circle the most expensive queries, read the plan with EXPLAIN, decide whether an index pays off, and spot the most common index-defeating patterns.Redis Slowdown Playbook: Big Keys, Blocking Commands and EvictionRedis processes commands on a single thread, so one blocking hotspot slows the whole instance. This guide follows the order "memory level → big keys → blocking commands → eviction and persistence", each step with copy-ready commands.Kubernetes CrashLoopBackOff: Diagnose a Restart-Looping Pod in 5 StepsCrashLoopBackOff is not one error but a state: the container dies on start, and Kubernetes retries with backoff. This guide gives the shortest path — exit code → previous logs → probes → resources → dependencies.SSH Passwordless Login: from ssh-keygen to Permission deniedPasswordless SSH boils down to one sentence: put the public key into the remote authorized_keys. What blocks you is usually the three hidden gates — file permissions, the home directory, and SELinux. This guide covers the full setup and a step-by-step debugging path.Python Virtual Environments in Practice: venv, Dependency Pinning and Cross-Platform Gotchas"Works on my machine" usually means mixed environments. This guide builds a clean flow with the stdlib venv: create, activate, pin and reproduce — plus the three frequent pitfalls: Windows activation, version drift and cross-platform dependencies.VSCode Remote Development: Remote-SSH Setup and Connection DebuggingA weak laptop, code on an intranet server, GPUs in the cluster — Remote-SSH keeps the UI local while code and terminal run remotely, feeling native. This guide covers setup, passwordless auth, common connection failures and experience tuning.Bash Script Safety: set -euo pipefail and trap for Trustworthy ScriptsA Bash script that "finishes" is not a script that "worked": failed commands do not stop it, empty variables pass silently, and pipes hide upstream crashes. This guide fixes all of them with a copy-ready skeleton (set -euo pipefail + trap + quoting discipline).YAML Gotchas: Type Coercion, Indentation and Multiline StringsYAML files that "look right but run wrong" usually fail in three places: silently coerced scalars, misplaced indentation, and ambiguous multiline strings. This guide dissects each trap with detection methods and a toolchain.HTTP Caching in Practice: Configuring Cache-Control and ETag CorrectlyMisconfigured caching ends one of two ways: users never see updates, or origin traffic explodes. This guide covers the split between freshness and validation caching, the hashed-asset vs HTML strategy divide, plus ready-to-use Nginx config and verification commands.Git Merge and Rebase Conflict Resolution in PracticeFor developers branching daily, this guide reads merge conflict markers end to end with reproducible cases and shows how to handle merge, rebase, and rename conflicts.Bisecting: Pinpoint the Commit That Introduced a BugFor anyone tired of hunting commits by gut feel, this is the binary-search way to find the commit that broke something, including making the check scriptable so bisect runs hands-free.Git Submodule Management and Its Common TrapsA field guide to when a submodule actually makes sense, how to add, clone, and update it, and the standard escape hatches for empty dirs after clone, branch drift, and nesting.Parallel Development with git worktree: Multiple Branches at OnceFor devs juggling several branches who fear stash chaos while switching, here is how to keep one clean worktree per task and never stash-swap again.docker compose Multi-Service Integration and Startup TroubleshootingGetting to the bottom of ordering, environment variables, port collisions, and service-name DNS inside the compose network, with copyable yaml and debugging commands.Kubernetes HPA: Configuring and Verifying AutoscalingFor engineers comfortable with Deployments but new to autoscaling: from installing metrics-server, writing an HPA, to load-testing and actually watching replicas scale out.Dockerfile Security Hardening: Shrinking the Attack SurfaceA landing checklist for image maintainers and service containerizers: non-root user, read-only root FS, dropping capabilities, layer hygiene, and image scanning.JavaScript Error Handling: From try/catch to Global Fallbacks and RetryA practical catalog for backend and full-stack JS: catching sync and async errors, Promise unhandledrejection, global fallbacks, and exponential-backoff retries.Python Packaging and Dependency Locking: From pyproject.toml to PyPIFor turning scripts into publishable packages: pyproject.toml for metadata, venv isolation, lock files for reproducible environments, and releasing to PyPI with twine.Linux Memory Exhaustion and OOM Hunting: From dmesg to the Process Likely Out of ControlWhen a server claims it is out of memory it rarely tells the whole truth; this guide shows how to use free, /proc/meminfo, cgroups, and dmesg to close in on the process actually eating memory, and how to configure against recurring OOM kills.Linux Network Troubleshooting Field Manual: Connectivity, Packet Loss, Latency, and DNSFrom unreachable pings to crawling slowness and flaky domain resolution, this guide walks layer by layer—link, IP, transport, application—showing which command and which field to look at, with copy-run experiments along the way.Linux File Permissions and sudo in Practice: chmod/ACL/setuid and Privilege BoundariesFrom bit permissions to ACLs to why setuid exists, this guide clarifies exactly how read/write/execute are computed, when the special bits backfire, and why sudo should not be a blanket ALL—plus a safer habit before you break something.Nginx Reverse Proxy in Practice: location Priority, proxy_pass, and Upstream Balancing Done RightMisconfigured proxies usually come down to location matching order and trailing-slash traps; this guide corrects the most commonly botched configs line by line and walks you through wiring a real Node backend into a load-balanced reverse proxy.HTTPS Certificates and Let's Encrypt Auto-Renewal: From Issuance to a cron FallbackManually requesting a cert is easy; the hard part is renewing it before expiry automatically. This guide walks certbot through webroot and HTTP-01 validation, configures renewal on port 80, and pairs it with two fallbacks so missing a renewal still does not take the site down.Debugging REST APIs End-to-End with curl: Methods, Headers, Auth, Upload, and Diagnosiscurl is the first scene of most API debugging; this guide starts from "see exactly what a request sent", covering GET/POST, query vs body, Bearer/JWT auth, Multipart uploads, and reading status code plus timing from one response.MySQL Logical Backup and Restore: mysqldump, Faster Imports, and Recovering from Accidental DeletesExporting with mysqldump is easy; the hard parts are whether the dump restores cleanly, whether the restores fast enough, and whether an accidental delete comes back. This guide covers consistency flags, single-transaction isolation, import-acceleration knobs, and a binlog-based recovery flow.PostgreSQL Slow-Query Diagnosis and Tuning: From EXPLAIN to shared_buffersSlow query tuning is a three-step ladder: find the query, read the plan, tune the config. This guide covers enabling the slow log, telling Seq Scan from Index Scan in EXPLAIN, and the core knobs that give the best bang for buck on buffers and concurrency.Cache Penetration, Breakdown and Avalanche: Identify and Protect Three Redis FaultsWhen your API suddenly slows down or your database collapses, three kinds of cache faults are usually to blame. This guide uses realistic log symptoms to help you tell apart penetration, breakdown and avalanche, then lays out practical defenses including mutex locks, null-caching and never-expiring hot keys.MySQL Index Design and EXPLAIN Analysis: From Wasteful to Zero Slow QueriesYou keep adding indexes yet slow queries do not go away — that usually means the indexes themselves are misdesigned. Starting from real business SQL, this guide covers composite index column order, covering indexes and index-failure scenarios, then walks through EXPLAIN output section by section so you can actually read a plan.tmux Session Management and Reuse: Remote Dev That Survives SSH DropsOne dropped SSH and a half-run build, open files and a dozen panes are all gone. This guide shows how tmux named sessions, detach/attach reuse and window/pane layout let you reconnect after a network drop and pick up exactly where you left off.VSCode Debugging Primer: From launch.json to Hovering Over a VariableCompared with sprinkling console.log everywhere, debugging lets you pause and inspect the call stack and every variable at each step. Starting from editing launch.json, this guide gets Node and front-end projects debugging, then walks through the sidebar, watch expressions and conditional breakpoints.Environment Variables and .env Management: Practical Details Beyond 12-FactorSecrets in code, environment variables drifting across environments, and .env committed to git — almost every project trips on these. Following the 12-factor principle, this guide clarifies precedence, injection methods, and how to manage local and CI config safely.Prettier and ESLint Together: Formatting and Linting That Do Not FightPrettier handles formatting and ESLint enforces quality, but when their rules overlap they fight each other and pre-commit runs get messy. This guide explains how to divide responsibilities, turn off conflicting rules, unify config, and enforce everything in CI.Take Claude Code Through a Real Bug-Fix FlowAn AI coding assistant is not "let it auto-fix" but a disciplined workflow. Using a real project bug, this guide walks the full loop from reproduction, adding a test, surgical fixes, to regression verification, with the exact instructions to issue at each step.Prompt Engineering for Coding Agents: Make Claude Code / Codex a Reliable CollaboratorGetting a coding agent right on the first try is not about the model but the instruction quality. This guide explains context constraints, minimal-change rules, test-first practice and review prompts, so you can craft reproducible prompts with fewer hallucinations.
← Back to Guides