Kubernetes Probe Tuning: How liveness and readiness Should Be Split

What each of the three probes actually controls

readinessProbe answers "can this pod receive traffic now". On failure the kubelet only removes the pod from Service endpoints; the container keeps running and is never restarted. During a rolling update it decides when a new pod joins traffic, which is what makes a deploy smooth.

livenessProbe answers "is this process irrecoverably broken". Once the failure threshold is hit, the kubelet kills the container and the restart policy recreates it. The bar is "no recovery without a restart", such as a deadlock or a stalled event loop. Transient dependency failures and short network blips must never gate liveness.

startupProbe answers "not up yet, do not pull the trigger". Until it succeeds, liveness and readiness are not evaluated at all. Slow-booting apps rely on it to avoid being killed mid-boot, which is the officially recommended replacement for inflating initialDelaySeconds.

readinessProbe:
  httpGet: { path: /readyz, port: 8080 }
  periodSeconds: 5
  failureThreshold: 3
livenessProbe:
  httpGet: { path: /healthz, port: 8080 }
  periodSeconds: 10
  failureThreshold: 3
startupProbe:
  httpGet: { path: /healthz, port: 8080 }
  periodSeconds: 5
  failureThreshold: 30   # 最长容忍 150 秒启动窗口

How to size the startup budget

The startup budget equals periodSeconds multiplied by failureThreshold, plus initialDelaySeconds before the first probe. For a 150 second tolerance, use 5 x 30. Only after startupProbe succeeds do the liveness and readiness timers begin counting.

Size the budget to cover the worst-case cold start: a JVM loading classes, opening database connections, running migrations. A load-tested cold start P99 is a reasonable baseline, but leave headroom because startup slows down under node pressure. An overly large budget only delays the restart of a genuinely stuck process.

A frequent mistake is pointing startupProbe and liveness at the same deep dependency endpoint. Once the startup budget is exhausted liveness takes over, and any dependency blip kills the container, producing a restart storm. Keep /healthz checking the process itself and let readiness carry dependency state.

startupProbe:
  httpGet: { path: /healthz, port: 8080 }
  periodSeconds: 5
  failureThreshold: 30
# 检查点：所有依赖都在 healthz 之外；依赖检查只出现在 /readyz
kubectl exec -it deploy/api -- curl -s localhost:8080/healthz

Timeouts and periods: where the defaults bite

timeoutSeconds defaults to just 1 second; a timeout is counted as a failure rather than an instant restart. If GC pauses or slow disks push handling time past 1 second, the health endpoint itself gets falsely marked unhealthy. For JVM services with visible GC pauses, raise it to 3 to 5 seconds.

periodSeconds defaults to 10 and failureThreshold to 3, so roughly 30 seconds of continuous failure is needed before a restart. Shorten the period to detect deadlocks sooner, but probing itself costs, especially exec probes which fork a process inside the container. Very high frequencies become a noise source under node pressure.

An exec probe spawns a process in the container on every check, so a heavy command is expensive. Avoid recursive finds or network calls inside it and prefer a lightweight HTTP or gRPC probe instead. successThreshold is fixed at 1 for liveness; only readiness allows a value above 1.

livenessProbe:
  exec:
    command: [cat, /tmp/healthy]
  timeoutSeconds: 5
  periodSeconds: 10
  failureThreshold: 3
  successThreshold: 1
readinessProbe:
  httpGet: { path: /readyz, port: 8080 }
  periodSeconds: 5
  successThreshold: 2   # 连续两次成功才接流量

A repeatable debugging path for probe failures

Start with events rather than speculation. `kubectl describe pod` prints the literal reason, such as Liveness probe failed: Get "http://10.244.0.5:8080/healthz": context deadline exceeded, and that message is more reliable than reasoning.

Connection refused usually means the listen address or port is wrong: the app binds only to 127.0.0.1, or the container port number is wrong. A 404 means the path is wrong or only registered inside a specific route group. Context deadline exceeded almost always means timeoutSeconds is too small.

Then reproduce the probe request from inside the container to separate "probe misconfigured" from "the app is genuinely broken". Compare the RESTARTS column: liveness restarting repeatedly while readiness only fails is a configuration problem; a readiness probe that succeeds then immediately fails again means you should read the application logs.

Finally, look at the cost of the probe endpoint itself. Plenty of services implement /healthz as a deep health check that scans a whole table or issues a database query, so a database hiccup times out every probe at once and amplifies one external failure into a fleet-wide restart. Keep each endpoint single-purpose and move deep diagnostics to an interface that humans call on demand.

kubectl describe pod -l app=api | sed -n "/Events:/,$ p"
kubectl get pods -l app=api -o wide
kubectl logs <pod> --previous --tail=100    # 看崩溃前最后一批日志
kubectl exec -it <pod> -- curl -sv localhost:8080/healthz

Pre-rollout verification checklist

First, the startup budget: multiply startupProbe periodSeconds by failureThreshold and confirm the product really exceeds the worst cold start from load testing, otherwise a slow-booting machine can still be killed before becoming ready.

Second, endpoint separation: confirm /healthz touches no external dependency, only /readyz checks dependencies, and that all three probe paths actually exist in the application. A wrong path is the most common mistake and the easiest one to catch before rollout.

Third, rollback verification: temporarily point livenessProbe at a path guaranteed to return 500 and confirm the pod restarts as expected, then restore the correct value. This proves the probe plumbing works before you need it, instead of discovering during an incident that the kubelet was never probing at all.

Fourth, load verification: watch readiness transitions while the node CPU is saturated. If readiness flaps at ordinary CPU usage, the combination of successThreshold and periodSeconds leaves too narrow a tolerance window, so widen it or adopt the progressive relaxation pattern that startupProbe represents.

# 快速验证探针链路是否真的生效
kubectl apply -f - <<YAML
apiVersion: v1
kind: Pod
metadata:
  name: probe-check
spec:
  containers:
  - name: app
    image: acme/api:1.4
    livenessProbe:
      httpGet: { path: /definitely-not-here, port: 8080 }
      periodSeconds: 5
      failureThreshold: 2
YAML
kubectl get pod probe-check -w          # 观察是否按预期重启
kubectl delete pod probe-check