巧匠Developer Cheatsheet
Containers & Orchestration

Kubernetes CrashLoopBackOff: Diagnose a Restart-Looping Pod in 5 Steps

CrashLoopBackOff is not one error but a state: the container dies on start, and Kubernetes retries with backoff. This guide gives the shortest path — exit code → previous logs → probes → resources → dependencies.

By 巧匠 Team·8 min read·Updated 2026-08-30

Step 1: read the exit code — it is a taxonomy

In `kubectl describe pod`, the Last State block has two key fields: Reason and Exit Code. The exit code is a ready-made taxonomy: 137 = killed by SIGKILL (usually OOMKilled); 143 = SIGTERM graceful exit (the app failed to finish within the grace period); 1 = the application errored out (go read the logs); 0 yet restarting = the process finished its job and exited (a one-shot task run as a long-lived service).

Classify first, then drill down — it avoids detours on the most common case, OOM, where the container log usually shows nothing abnormal.

kubectl describe pod <pod> -n <ns>
# Last State: Terminated, Reason: OOMKilled, Exit Code: 137

Step 2: read the logs of the previous crash

Once the container restarts, plain `kubectl logs` shows only the current (short, often empty) instance. Add `--previous` for the real crash-site output: missing config, unreachable database, port conflicts — startup-phase errors live there.

For multi-container Pods, pass `-c <container>`. When the app logs to a file instead of stdout (common with Tomcat-style Java apps), fix the log output or collect via a sidecar, otherwise kubectl logs will never see it.

kubectl logs <pod> -n <ns> --previous
kubectl logs <pod> -n <ns> -c <container> --previous

Step 3: check the probes — friendly fire is common

A failing liveness probe restarts the container, looking exactly like a real crash — describe shows the probe as the Reason (e.g. Liveness probe failed). Typical misconfigurations: liveness wired to a hard dependency (slow-starting apps judged dead before ready), initialDelaySeconds too short, wrong probe port/path.

Principle: liveness answers "is it dead", readiness answers "can it take traffic". For slow starters add a startupProbe or raise initialDelaySeconds, separating "still booting" from "actually dead".

livenessProbe:
  httpGet:
    path: /healthz
    port: 8080
  initialDelaySeconds: 30   # 启动慢的应用别配太短
  periodSeconds: 10

Step 4: resource limits — two sources of OOMKilled

Exceeding the memory limit gets the container killed by the kernel (exit code 137). Check two places: ① the limit itself is too small — for Java, modern JVMs are cgroup-aware (JDK 8u191+ / JDK 10+), but a hand-set -Xmx above the limit still gets killed; ② an app memory leak — shows as OOM after running a while; the container memory curve distinguishes "pegged from the start" from "slow climb".

Raising the limit is a stopgap; leaks need a heap dump on the app side. A wide requests/limits gap also invites node overcommit and eviction — do not look at the container alone.

resources:
  requests: { memory: "256Mi", cpu: "100m" }
  limits:   { memory: "512Mi" }

Step 5: dependencies not ready — the ordering problem

The app connects to a database/Redis at boot, crashes on refusal, and retries — the log shows a clean connection refused. This CrashLoopBackOff is really an overly hard retry policy. Instead of relying on Kubernetes restarts, add app-side retry with backoff, or block with initContainers (a wait-for-db script) before the main container starts.

Projects ported from Docker Compose hit this most: compose has depends_on + condition; Kubernetes has no direct equivalent, so you must supply the readiness logic yourself.

initContainers:
  - name: wait-for-db
    image: busybox
    command: ["sh", "-c", "until nc -z db-svc 5432; do sleep 2; done"]

Wrap-up: a quick reference

describe for the exit code (137=OOM / 1=error / 0=clean exit yet restarting) → logs --previous for the crash scene → rule out probe kills → verify limits against the memory curve → add dependency-readiness logic. Most CrashLoopBackOffs close within these five steps; whatever remains is usually an image bug — reproduce locally with the same image.

kubectl describe pod <pod> -n <ns>   # 1 退出码
kubectl logs <pod> -n <ns> --previous   # 2 崩溃现场

More guides

Git Accident Recovery: Fix Wrong Commits, Wrong Branches and Deleted BranchesThe scariest moment in daily work is "oh no, I messed up a commit or branch". This guide gives copy-ready recovery commands for real accident scenarios, with clear safety boundaries for each.Docker Image Slimming: 5 Steps from 1.2GB to 120MBBigger images mean slower pulls, larger attack surface and higher storage cost. This guide gives 5 immediately applicable slimming steps, each with a reusable Dockerfile snippet and a verification command.Nginx 502 Bad Gateway? Locate It in 5 StepsA 502 means Nginx itself is fine but got an invalid response from the upstream (backend/proxy). This guide gives the shortest path: process → config → timeout → resources → logs.Linux Disk Full: Locate the Real Culprit with df, du and lsofA "No space left on device" error does not mean you should delete files at random. This guide walks through the full path — filesystem water level, directory drill-down, and "deleted but still held" files — plus a safe cleanup order that will not destroy data.Linux CPU at 100%: the Shortest Path from top to the Thread Stack"Restart fixes everything" is the most expensive option when CPU is pegged. This guide gives a reusable path: classify the load, pin the thread, capture what it is executing, then act — with dedicated tools for Java, Python and Go.MySQL Slow Query Tuning: Slow Log, EXPLAIN and Indexing"The database is slow" usually means a handful of slow SQL statements. This guide offers a repeatable flow: use the slow log to circle the most expensive queries, read the plan with EXPLAIN, decide whether an index pays off, and spot the most common index-defeating patterns.Redis Slowdown Playbook: Big Keys, Blocking Commands and EvictionRedis processes commands on a single thread, so one blocking hotspot slows the whole instance. This guide follows the order "memory level → big keys → blocking commands → eviction and persistence", each step with copy-ready commands.SSH Passwordless Login: from ssh-keygen to Permission deniedPasswordless SSH boils down to one sentence: put the public key into the remote authorized_keys. What blocks you is usually the three hidden gates — file permissions, the home directory, and SELinux. This guide covers the full setup and a step-by-step debugging path.Python Virtual Environments in Practice: venv, Dependency Pinning and Cross-Platform Gotchas"Works on my machine" usually means mixed environments. This guide builds a clean flow with the stdlib venv: create, activate, pin and reproduce — plus the three frequent pitfalls: Windows activation, version drift and cross-platform dependencies.VSCode Remote Development: Remote-SSH Setup and Connection DebuggingA weak laptop, code on an intranet server, GPUs in the cluster — Remote-SSH keeps the UI local while code and terminal run remotely, feeling native. This guide covers setup, passwordless auth, common connection failures and experience tuning.
← Back to Guides