Match Three Real Production Symptoms to the Fault
Teams often blur the three faults together, but they can be told apart quickly from observable behavior. Cache penetration means the requested data exists in neither the cache nor the database, so every request drills through to the database. Cache breakdown happens when one hot key expires and a wave of concurrent requests all try to rebuild it. Cache avalanche occurs when a large number of keys expire en masse around the same moment, flattening the database instantly.
Separate them by metrics first: penetration pushes database read QPS abnormally high while the hit rate drops, but responses are mostly empty data. Breakdown shows the latency of a single hot key spiking suddenly, with errors concentrated on a handful of endpoints. Avalanche hits several endpoints at once with waterfall-like peaks. Inspect the shape of the cache hit-rate curve before guessing.
redis-cli -n 0 info keyspace
# ^-- keyspace 看命中率与过期 key 总数
redis-cli -n 0 info stats | grep -E "expired_keys|keyspace_hits|keyspace_misses"Add a Bloom Filter to Block Keys That Do Not Exist
Penetration is usually caused by invalid parameters or scanning of non-existent IDs. When you place a Bloom filter between the cache and the database, it can judge whether a key might exist; a "does not exist" verdict lets you return empty right away, blocking most forged requests.
A Bloom filter has a false-positive rate but never a false-negative: it may claim a key exists (when it does not), but it will never claim an existing key is absent — so its "absent" verdict is trustworthy. Its job is to block keys that clearly do not exist; keys that pass the filter then continue to cache and database. The simplest route is Redis’s bloom module (BF.ADD / BF.EXISTS), or you can keep one in-process with Google’s guava BloomFilter.
redismodule-dev
redis-cli BF.ADD blacklist 1000001
redis-cli BF.EXISTS blacklist 1000001 # 1
redis-cli BF.EXISTS blacklist 5030304 # 0 不存在，直接挡
# 应用层配合：命中过滤器且查库为空 -> 给 key 写入一个短 TTL 的空值缓存Rebuild Breakdown under a Mutex Lock Instead of Letting Every Request Hit the Source
The instant a hot key expires, if dozens of threads all find the cache empty they all race to the database — that is breakdown. The most common fix is a mutex lock: on a cache miss, allow only one request to rebuild while others wait, then read the cache once it is populated.
Two caveats when using Redis SET NX as a distributed lock: always attach an expiration so a crashed holder cannot deadlock, and keep the window between locking and rebuilding short. In Go, only the goroutine whose SetNX succeeds actually queries the database and refills; the others spin briefly and then read the cache. An alternative is "never expire plus background refresh": hot keys get no TTL, a scheduled job refreshes them periodically, and the expiration window disappears entirely.
func Get(key string) (string, error) {
if v, err := rdb.Get(ctx, key).Result(); err == nil {
return v, nil
}
// 抢锁：SET key 1 EX 5 NX，抢到才回源
ok, _ := rdb.SetNX(ctx, "lock:"+key, 1, 5*time.Second).Result()
if !ok {
time.Sleep(50 * time.Millisecond) // 未抢到，稍候再读
return rdb.Get(ctx, key).Result()
}
defer rdb.Del(ctx, "lock:"+key) // 回填完释放锁
v := loadFromDB(key)
rdb.Set(ctx, key, v, 60*time.Second)
return v, nil
}Avalanche Comes from Homogeneous Expiry — Shake It Up
Avalanche is rarely one broken key; it is usually the cache giving every key the same TTL so everything expires at once. The fix is to break expiry homogeneity: add a random offset to each TTL so keys expire at staggered times.
For example, with a 30-minute base TTL you would generate a random value between 30 and 35 minutes per key. For keys that demand high availability, consider "permanent plus async dual-write": the primary key never expires and a sidecar job quietly refreshes it during low-traffic windows. On the database side, keep degradation ready: circuit breakers, rate limiting and read replicas to spread the load so a sudden surge cannot flatten the database in one pass.
const base = 60 * 30 // 基础 30 分钟
const jitter = 60 * int(rand.Float64()*300) // 0-5 分钟随机偏移
rdb.Set(ctx, key, v, time.Duration(base+jitter)*time.Second)Verify That All Three Layers of Defense Actually Work
Do not just assume things are fine after configuring. Build three cheap checks: for penetration, hammer a batch of non-existent IDs and confirm database read QPS no longer spikes; for breakdown, set a hot key, manually DEL it, then fire concurrent requests and confirm the hit rate stays stable; for avalanche, temporarily give every key the same TTL and watch the expiry curve spread out.
Redis’s monitor command shows actual source-refresh count most directly: under normal cache hits those commands never appear, so a burst of database-rebuild commands means your defense is not holding. Make the keyspace_hits/keyspace_misses ratio, database read QPS and the core endpoint P99 latency your three alert lines — whenever any one of them is off, re-check the cache layer first.
redis-cli -n 0 monitor # 观察实时命令，若瞬间出现大量 db 重建命令说明防护失效
redis-cli -n 0 info stats | grep -E "keyspace_hits|keyspace_misses" | awk -F: '{print $1": "$2}'