巧匠Developer Cheatsheet
Containers & Orchestration

Kubernetes HPA: Configuring and Verifying Autoscaling

For engineers comfortable with Deployments but new to autoscaling: from installing metrics-server, writing an HPA, to load-testing and actually watching replicas scale out.

By LaoHand Team·8 min read·Updated 2026-09-06

Prerequisite: Metrics-Server Before HPA Has Anything to Compute

HPA computes desired replicas from metrics, which come from metrics-server (CPU/memory) or custom/external sources. Without metrics-server, the HPA object exists but always shows <unknown> and never scales.

kind and minikube sometimes ship without it; install explicitly. Prove readiness by kubectl top nodes and kubectl top pods returning numbers.

metrics-server samples the kubelet summary API every ~15s; mind image mirroring and TLS flags in restrictive environments before tuning HPA.

# 安装 metrics-server（最新 release 见其 GitHub）
kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml

# 确认采集正常
kubectl top nodes
kubectl top pods

kubectl get apiservices | grep metrics

Write a First HPA: CPU-Based with min/max and Threshold

HPA uses averages: it sums Pod CPU, divides by their requests to get a utilization %, and scales out past the target (e.g. 60%) and back in when clearly below it for a sustained window.

Key point: the Deployments must set resources.requests or HPA has no denominator. The target may be an absolute value, but utilization % is the common choice.

Cold-start and jitter matter: scale-out can jump straight to max, while scale-down respects a stabilizationWindowSeconds (default 300s). Keep min at least 1-2 for availability, max within budget.

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: api-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: api
  minReplicas: 2
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 60

# Deployment 侧必须有 requests：
#   resources:
#     requests: { cpu: 250m, memory: 256Mi }

Mistake: HPA Stuck on <unknown>, Never Scaling

The classic symptom: kubectl get hpa shows <unknown>/-- under TARGETS and replicas never move. Causes in order: metrics-server missing or not authorized, Deployment lacking resources.requests, namespace resource quotas, or a misnamed custom metric.

Diagnose fast: kubectl describe hpa api-hpa shows conditions plus an "unable to retrieve metrics" message; then kubectl top pods to see whether CPU data even exists in the namespace.

Fix in order: make top return numbers (fix metrics-server), ensure requests exist, only then suspect the HPA yaml itself. Do not rewrite yaml endlessly while ignoring the data source.

# 症状
kubectl get hpa
# NAME      REFERENCE   TARGETS   MINPODS   MAXPODS   REPLICAS
# api-hpa   Deployment/api  <unknown>/60%  2  10  2

# 看根因
kubectl describe hpa api-hpa
# ... Unable to retrieve metrics for resource cpu ...

kubectl top pods -n <ns>
# 空或 error: metrics not available yet  -> 先修 metrics-server

Verify It Truly Scales: Load It to Blow Past the Threshold

Configuring without verifying is the same as not configuring. The most direct check: push temporary CPU load onto one Pod in the Deployment and watch HPA pull replicas from min upward in minutes.

Use busybox with a CPU-burning loop, or kubectl run a `while :; do :; done` under resource limits and watch. Just do not saturate the cluster so far that it can neither sprawl nor recover.

Clean up afterward: delete the loader Pod and drop the Deployment back to min (or clean the HPA env), so test load does not sit in production eating resources.

# 给目标 Deployment 派一个 CPU 燃烧 Pod
kubectl run loader \
  --image=busybox \
  --restart=Never \
  --requests='cpu=300m' \
  -- sh -c "while true; do :; done"

# 观察扩展（等待 stabilization 后）
kubectl get hpa api-hpa -w
kubectl get pods -w

# 收尾
kubectl delete pod loader
kubectl scale deployment api --replicas=2

Going Further: Multi and Custom Metrics (Memory, External Queue)

CPU alone is blind to I/O-bound or batch work; add a memory metric, either Utilization against the request or an absolute Value in MB.

Queue-backlog scaling (Kafka lag, Celery task count) is the canonical external-metric case and needs an external-metric adapter such as KEDA.

With several metrics HPA takes the maximum of the per-metric desired values, so a low-scoring metric never hides a high-pressure one.

spec:
  metrics:
  - type: Resource
    resource:
      name: cpu
      target: { type: Utilization, averageUtilization: 60 }
  - type: Resource
    resource:
      name: memory
      target: { type: Utilization, averageUtilization: 75 }
  # 或外部队列积压（经 KEDA / external adapter）
  # - type: External
  #   external:
  #     metric:
  #       name: kafka_lag
  #     target: { type: AverageValue, averageValue: "20" }

Official References

Each command links to its official documentation below, so you can verify the latest usage and read deeper.

More guides

Git Accident Recovery: Fix Wrong Commits, Wrong Branches and Deleted BranchesThe scariest moment in daily work is "oh no, I messed up a commit or branch". This guide gives copy-ready recovery commands for real accident scenarios, with clear safety boundaries for each.Docker Image Slimming: 5 Steps from 1.2GB to 120MBBigger images mean slower pulls, larger attack surface and higher storage cost. This guide gives 5 immediately applicable slimming steps, each with a reusable Dockerfile snippet and a verification command.Nginx 502 Bad Gateway? Locate It in 5 StepsA 502 means Nginx itself is fine but got an invalid response from the upstream (backend/proxy). This guide gives the shortest path: process → config → timeout → resources → logs.Linux Disk Full: Locate the Real Culprit with df, du and lsofA "No space left on device" error does not mean you should delete files at random. This guide walks through the full path — filesystem water level, directory drill-down, and "deleted but still held" files — plus a safe cleanup order that will not destroy data.Linux CPU at 100%: the Shortest Path from top to the Thread Stack"Restart fixes everything" is the most expensive option when CPU is pegged. This guide gives a reusable path: classify the load, pin the thread, capture what it is executing, then act — with dedicated tools for Java, Python and Go.MySQL Slow Query Tuning: Slow Log, EXPLAIN and Indexing"The database is slow" usually means a handful of slow SQL statements. This guide offers a repeatable flow: use the slow log to circle the most expensive queries, read the plan with EXPLAIN, decide whether an index pays off, and spot the most common index-defeating patterns.Redis Slowdown Playbook: Big Keys, Blocking Commands and EvictionRedis processes commands on a single thread, so one blocking hotspot slows the whole instance. This guide follows the order "memory level → big keys → blocking commands → eviction and persistence", each step with copy-ready commands.Kubernetes CrashLoopBackOff: Diagnose a Restart-Looping Pod in 5 StepsCrashLoopBackOff is not one error but a state: the container dies on start, and Kubernetes retries with backoff. This guide gives the shortest path — exit code → previous logs → probes → resources → dependencies.SSH Passwordless Login: from ssh-keygen to Permission deniedPasswordless SSH boils down to one sentence: put the public key into the remote authorized_keys. What blocks you is usually the three hidden gates — file permissions, the home directory, and SELinux. This guide covers the full setup and a step-by-step debugging path.Python Virtual Environments in Practice: venv, Dependency Pinning and Cross-Platform Gotchas"Works on my machine" usually means mixed environments. This guide builds a clean flow with the stdlib venv: create, activate, pin and reproduce — plus the three frequent pitfalls: Windows activation, version drift and cross-platform dependencies.VSCode Remote Development: Remote-SSH Setup and Connection DebuggingA weak laptop, code on an intranet server, GPUs in the cluster — Remote-SSH keeps the UI local while code and terminal run remotely, feeling native. This guide covers setup, passwordless auth, common connection failures and experience tuning.Bash Script Safety: set -euo pipefail and trap for Trustworthy ScriptsA Bash script that "finishes" is not a script that "worked": failed commands do not stop it, empty variables pass silently, and pipes hide upstream crashes. This guide fixes all of them with a copy-ready skeleton (set -euo pipefail + trap + quoting discipline).YAML Gotchas: Type Coercion, Indentation and Multiline StringsYAML files that "look right but run wrong" usually fail in three places: silently coerced scalars, misplaced indentation, and ambiguous multiline strings. This guide dissects each trap with detection methods and a toolchain.HTTP Caching in Practice: Configuring Cache-Control and ETag CorrectlyMisconfigured caching ends one of two ways: users never see updates, or origin traffic explodes. This guide covers the split between freshness and validation caching, the hashed-asset vs HTML strategy divide, plus ready-to-use Nginx config and verification commands.Git Merge and Rebase Conflict Resolution in PracticeFor developers branching daily, this guide reads merge conflict markers end to end with reproducible cases and shows how to handle merge, rebase, and rename conflicts.Bisecting: Pinpoint the Commit That Introduced a BugFor anyone tired of hunting commits by gut feel, this is the binary-search way to find the commit that broke something, including making the check scriptable so bisect runs hands-free.Git Submodule Management and Its Common TrapsA field guide to when a submodule actually makes sense, how to add, clone, and update it, and the standard escape hatches for empty dirs after clone, branch drift, and nesting.Parallel Development with git worktree: Multiple Branches at OnceFor devs juggling several branches who fear stash chaos while switching, here is how to keep one clean worktree per task and never stash-swap again.docker compose Multi-Service Integration and Startup TroubleshootingGetting to the bottom of ordering, environment variables, port collisions, and service-name DNS inside the compose network, with copyable yaml and debugging commands.Dockerfile Security Hardening: Shrinking the Attack SurfaceA landing checklist for image maintainers and service containerizers: non-root user, read-only root FS, dropping capabilities, layer hygiene, and image scanning.JavaScript Error Handling: From try/catch to Global Fallbacks and RetryA practical catalog for backend and full-stack JS: catching sync and async errors, Promise unhandledrejection, global fallbacks, and exponential-backoff retries.Python Packaging and Dependency Locking: From pyproject.toml to PyPIFor turning scripts into publishable packages: pyproject.toml for metadata, venv isolation, lock files for reproducible environments, and releasing to PyPI with twine.Linux Memory Exhaustion and OOM Hunting: From dmesg to the Process Likely Out of ControlWhen a server claims it is out of memory it rarely tells the whole truth; this guide shows how to use free, /proc/meminfo, cgroups, and dmesg to close in on the process actually eating memory, and how to configure against recurring OOM kills.Linux Network Troubleshooting Field Manual: Connectivity, Packet Loss, Latency, and DNSFrom unreachable pings to crawling slowness and flaky domain resolution, this guide walks layer by layer—link, IP, transport, application—showing which command and which field to look at, with copy-run experiments along the way.Debugging systemd Service Startup Failures: From status and Dependencies to Ordering Deadlockssystemctl is-failed is just the first step; this guide splits failures into four kinds—launcher error, process instantly exiting, dependency not ready, and port never listening—with the right log entry and fix for each.Linux File Permissions and sudo in Practice: chmod/ACL/setuid and Privilege BoundariesFrom bit permissions to ACLs to why setuid exists, this guide clarifies exactly how read/write/execute are computed, when the special bits backfire, and why sudo should not be a blanket ALL—plus a safer habit before you break something.Nginx Reverse Proxy in Practice: location Priority, proxy_pass, and Upstream Balancing Done RightMisconfigured proxies usually come down to location matching order and trailing-slash traps; this guide corrects the most commonly botched configs line by line and walks you through wiring a real Node backend into a load-balanced reverse proxy.HTTPS Certificates and Let's Encrypt Auto-Renewal: From Issuance to a cron FallbackManually requesting a cert is easy; the hard part is renewing it before expiry automatically. This guide walks certbot through webroot and HTTP-01 validation, configures renewal on port 80, and pairs it with two fallbacks so missing a renewal still does not take the site down.Debugging REST APIs End-to-End with curl: Methods, Headers, Auth, Upload, and Diagnosiscurl is the first scene of most API debugging; this guide starts from "see exactly what a request sent", covering GET/POST, query vs body, Bearer/JWT auth, Multipart uploads, and reading status code plus timing from one response.MySQL Logical Backup and Restore: mysqldump, Faster Imports, and Recovering from Accidental DeletesExporting with mysqldump is easy; the hard parts are whether the dump restores cleanly, whether the restores fast enough, and whether an accidental delete comes back. This guide covers consistency flags, single-transaction isolation, import-acceleration knobs, and a binlog-based recovery flow.PostgreSQL Slow-Query Diagnosis and Tuning: From EXPLAIN to shared_buffersSlow query tuning is a three-step ladder: find the query, read the plan, tune the config. This guide covers enabling the slow log, telling Seq Scan from Index Scan in EXPLAIN, and the core knobs that give the best bang for buck on buffers and concurrency.Cache Penetration, Breakdown and Avalanche: Identify and Protect Three Redis FaultsWhen your API suddenly slows down or your database collapses, three kinds of cache faults are usually to blame. This guide uses realistic log symptoms to help you tell apart penetration, breakdown and avalanche, then lays out practical defenses including mutex locks, null-caching and never-expiring hot keys.MySQL Index Design and EXPLAIN Analysis: From Wasteful to Zero Slow QueriesYou keep adding indexes yet slow queries do not go away — that usually means the indexes themselves are misdesigned. Starting from real business SQL, this guide covers composite index column order, covering indexes and index-failure scenarios, then walks through EXPLAIN output section by section so you can actually read a plan.tmux Session Management and Reuse: Remote Dev That Survives SSH DropsOne dropped SSH and a half-run build, open files and a dozen panes are all gone. This guide shows how tmux named sessions, detach/attach reuse and window/pane layout let you reconnect after a network drop and pick up exactly where you left off.VSCode Debugging Primer: From launch.json to Hovering Over a VariableCompared with sprinkling console.log everywhere, debugging lets you pause and inspect the call stack and every variable at each step. Starting from editing launch.json, this guide gets Node and front-end projects debugging, then walks through the sidebar, watch expressions and conditional breakpoints.Environment Variables and .env Management: Practical Details Beyond 12-FactorSecrets in code, environment variables drifting across environments, and .env committed to git — almost every project trips on these. Following the 12-factor principle, this guide clarifies precedence, injection methods, and how to manage local and CI config safely.Prettier and ESLint Together: Formatting and Linting That Do Not FightPrettier handles formatting and ESLint enforces quality, but when their rules overlap they fight each other and pre-commit runs get messy. This guide explains how to divide responsibilities, turn off conflicting rules, unify config, and enforce everything in CI.Take Claude Code Through a Real Bug-Fix FlowAn AI coding assistant is not "let it auto-fix" but a disciplined workflow. Using a real project bug, this guide walks the full loop from reproduction, adding a test, surgical fixes, to regression verification, with the exact instructions to issue at each step.Prompt Engineering for Coding Agents: Make Claude Code / Codex a Reliable CollaboratorGetting a coding agent right on the first try is not about the model but the instruction quality. This guide explains context constraints, minimal-change rules, test-first practice and review prompts, so you can craft reproducible prompts with fewer hallucinations.
← Back to Guides