First, separate the three problems: different causes, different fixes
A cache breakdown (often called a hot key expiring) happens when a single extremely popular key happens to expire at one instant. If rebuilding that key is expensive — aggregating several tables or calling an external service — every concurrent request discovers the empty cache at the same moment and hundreds of them hit the database together. The database is not genuinely broken, but it momentarily absorbs read traffic the cache was supposed to serve, the connection pool saturates, and latency jumps from milliseconds to seconds.
Cache penetration is different: the query targets a key that simply does not exist. Since the key was never cached, the cache can never hit, the database is never warmed, and every such request walks the full path of cache lookup, miss, database query. An attacker can generate a sequence of random IDs against your endpoint and hammer the database without ever producing a cache hit. It has nothing to do with concurrency or overall traffic volume — only with the fact that the key is absent.
A cache avalanche is a mass invalidation event. If you bulk-load a batch of entries at 02:00 with a TTL of 3600 seconds, they all expire together at 03:00 and a flood of traffic reaches the database in the same second. The other form is the Redis instance itself going down or losing network connectivity: all traffic has nowhere to go, which looks identical to an avalanche even though the cause is storage unavailability rather than colliding expiry times.
Learn the triage signature of each one, because it saves enormous time during an incident. A breakdown shows a narrow, tall spike in database QPS while the hot key hit rate momentarily drops to zero. A penetration shows steadily rising database QPS without a corresponding rise in total request volume, many empty results, and a Redis keyspace that barely grows. An avalanche shows hit rate falling off a cliff, with the timing lining up with a bulk write or an instance event.
# (1) Breakdown: a burst of high QPS on one key + its hit rate drops to zero
redis-cli --no-raw OBJECT FREQ "hot:config:global" # LFU counter, used to spot hot keys
redis-cli --no-raw INFO stats | grep -E "keyspace_(hits|misses)"
redis-cli --no-raw TTL "hot:config:global" # Remaining time to live
# (2) Penetration: hit rate normal/high, but the database sees many "not found" queries
redis-cli --no-raw DBSIZE # Watch whether the keyspace grows with traffic
redis-cli --no-raw SLOWLOG GET 10 # Locate the slow queries hit by penetration
# (3) Avalanche: hit rate drops across the board, the DB sees batches of identical slow queries
redis-cli --no-raw INFO memory | grep -E "used_memory_human|maxmemory_policy"
redis-cli --no-raw INFO persistence | grep -E "rdb_last_bgsave_status|aof_last_write_status"
redis-cli --no-raw INFO replication | grep -E "role|master_link_status"Taming breakdowns: mutex rebuilds and logical expiry
The most direct fix is a mutex around the rebuild: on a cache miss, first try SET lock:key token NX EX 10. The winner rebuilds, and everyone else either retries the cache after a short wait or returns a degraded value. However many requests arrive, exactly one reaches the database. The lock value must be a unique identifier such as a UUID, and unlocking must go through a Lua script that compares the token before deleting, otherwise a holder whose lock already expired will delete the lock acquired by a later owner.
Waiters should not sleep for a fixed duration. A common implementation spins a few times with short intervals, re-reading the cache each time and returning immediately on a hit. After a bounded number of retries with no hit, decide based on business tolerance whether to return a stale value, a default, or pass through to the database. Keep the retry count and interval configurable, because sensible values depend entirely on rebuild cost and traffic volume — constants copied from another project rarely fit.
Logical expiry, often called soft expire, takes a different route and sidesteps waiting altogether: the cache entry stores both the value and a logical expiry timestamp. On read, if the timestamp has passed, return the possibly stale value immediately and kick off a rebuild in the background. Readers are never blocked; the cost is that callers may observe data that is at most a bounded interval old.
Logical expiry is naturally immune to breakdowns because expiry detection and rebuild initiation are decoupled: no matter how many readers simultaneously see the entry as expired, only one rebuild really runs. The price is that you must define a business-level tolerance for staleness and require callers to accept occasional inconsistency. Product pages and home-page recommendations — read-heavy, rarely written, fine with a few seconds of lag — are ideal fits. Balances and stock levels are not.
# Mutex rebuild: acquire lock -> rebuild -> release lock (Node-style pseudocode, semantics match redis-cli)
# 1) Acquire the lock; NX guarantees a single winner, EX guarantees the lock auto-releases if the holder crashes
redis-cli SET "lock:user:1001" "b1f2c0de-uuid" NX EX 10
# OK means the lock was acquired, nil means someone else is already rebuilding
# 2) Waiters: spin at short intervals, first check whether the cache was already filled by someone else
redis-cli GET "cache:user:1001"
redis-cli EXISTS "lock:user:1001"
# 3) Releasing the lock must use Lua to keep "compare + delete" atomic,
# otherwise a timed-out holder will delete the new holder's lock by mistake
redis-cli EVAL "if redis.call(\"get\",KEYS[1])==ARGV[1] then return redis.call(\"del\",KEYS[1]) else return 0 end" 1 "lock:user:1001" "b1f2c0de-uuid"
# 4) Logical expiry: store a timestamp in the value, keep serving it after expiry, and rebuild in the background
redis-cli SET "cache:user:1001" "{\"data\":{\"name\":\"Alice\"},\"expireAt\":1758000000000}" EX 300Stopping penetration: null-value caching and Bloom filters
Null-value caching is the cheapest fix with the fastest payoff. When the database lookup returns nothing, do not report a miss and let the next request fall through again — store the fact that the key is absent, with a very short TTL such as 30 to 60 seconds. A scanner hitting random IDs then penetrates the database at most once per key, and even a sustained attack is throttled by that TTL.
The TTL for negative entries should be short, which is the opposite of the instinct used for positive entries. Real cached data may stay valid for minutes or hours, whereas a judgement of nonexistence ages quickly: a stale negative entry means genuinely existing data stays invisible far too long. Prefer leaking a few lookups over caching emptiness for long.
When negative caching is not enough — attackers using identifiers that are structurally valid but semantically dead, or a key space so large that the negative entries themselves cost real memory — reach for a Bloom filter. It is a bit vector plus several hash functions, into which every known-existing key is mapped ahead of time. A query consults the filter first: if the relevant bits are zero the key definitely does not exist, and you can reply without even writing a negative entry.
Two properties define the trade-off. A Bloom filter has no false negatives, only false positives: saying "absent" is always safe, saying "present" may just be a collision, so the real cache or database lookup still happens. Second, the classic structure cannot delete entries; deletion needs a counting variant such as a Cuckoo Filter, which costs more memory. Populating the filter is tightly coupled to the write path, and a missed insertion produces the catastrophic "exists but reported absent" failure, so populate on creation rather than backfilling later.
# Null-value caching (negative cache): give non-existent keys a short TTL
redis-cli SET "cache:article:999999" "__nil__" EX 45
redis-cli GET "cache:article:999999"
# Bloom filter: use EXISTS instead of probing one by one; the standard BF supports the FEXISTS extension
redis-cli BF.RESERVE "bf:article_id" 0.01 1000000
redis-cli BF.ADD "bf:article_id" "1001"
redis-cli BF.EXISTS "bf:article_id" "1001" # Returns 1: may exist, keep checking cache/database
redis-cli BF.EXISTS "bf:article_id" "999999" # Returns 0: definitely absent, return an empty result directly
redis-cli BF.INFO "bf:article_id" # Inspect capacity, error rate, and element count
# Check order: Bloom filter -> cache -> negative cache -> databasePreventing avalanches: TTL jitter and graceful degradation
The cheapest and most effective defence against an avalanche is TTL jitter. Add a random offset to the base expiry, for example 600 plus a random 0 to 300 seconds. A batch of ten thousand entries then expires across a ten-minute window instead of detonating in the same second. A jitter ratio of 10 to 20 percent of the base is normally plenty; pushing it further weakens the hit rate for no benefit.
The second defence is not betting everything on one instance. Set a memory ceiling and an eviction policy so Redis discards old keys under pressure. If the cache regularly runs near its memory limit, a restart does far less damage, because the node has already been practising eviction in steady state. Note that allkeys-lru puts hot keys into the eviction pool as well, trading hit rate for survival, whereas volatile-lru only evicts keys that carry a TTL — the better choice when every cache entry is supposed to have one.
The third defence is degradation inside the application. When Redis is genuinely unavailable, the app must hold on its own: cap the concurrency of fallthrough requests with a semaphore so only a few dozen reach the database, briefly cache failed lookups as negative results, and if necessary return the last successful response or a static fallback page. The critical property is that a cache outage must not surface as an exception; a 500 caused by a Redis timeout is far worse than simply reading through to the database.
Finally, preventing avalanches means preventing their creation. Offset the TTLs of a bulk warm-up so entries do not align; run a keyspace-growth estimate in a staging environment before a release to see how many cache keys a dataset will create, and treat a large number during peak traffic as a risk to be reviewed. An easily overlooked equivalent is a maintenance script that flushes the cache for testing — that has exactly the same blast radius as an avalanche, so move it to an off-peak slot and run it in batches.
# TTL jitter: add randomness to the base value so a batch of keys does not expire at once
# shell pseudocode: base=600, jitter=120
for key in $(cat .ai-temp/reports/hot_keys.txt); do
ttl=$(( base + RANDOM % (base / 5) ))
redis-cli SETEX "$key" "$ttl" "$(cat .ai-temp/downloads/payload.json)" > /dev/null
done
# Memory limit and eviction policy: keep the instance evicting even under normal load
redis-cli CONFIG SET maxmemory 2gb
redis-cli CONFIG SET maxmemory-policy volatile-lru
redis-cli CONFIG GET maxmemory-policy
redis-cli --no-raw INFO memory | grep -E "used_memory_human|maxmemory_human|mem_fragmentation_ratio"
redis-cli --no-raw INFO stats | grep evicted_keys
# Watch the hit rate: if it stays below 90% for long, revisit the TTL and key design
redis-cli CONFIG RESETSTAT
redis-cli --no-raw INFO stats | grep -E "keyspace_hits|keyspace_misses"Hot keys: a second-level local cache and request spreading
When traffic on a single key is high enough that one instance CPU-bound or network-bound, the cache itself becomes the problem. Mutexes do not help here, because the bottleneck sits on the Redis side rather than on the fallthrough path. The first tool is a second-level local cache inside the application process: a very small in-memory cache with a short expiry of a few to a few dozen seconds holding a copy of the hot value. Hot-key responses are usually not millisecond-sensitive, so seconds of staleness buy you the elimination of most read traffic at the process level.
Be deliberate about consistency and memory when adding a local tier. The standard pattern is double-checked access: look locally, and on a miss read Redis and repopulate the local entry. On write, actively invalidate local copies, or rely on a very short TTL to expire naturally, so instances do not diverge for long. The local cache must also have a hard capacity ceiling — typically an LRU with a maximum entry count — otherwise a workload of mostly unique keys turns it into a memory leak.
The second tool is spreading. Split a hot key into N replicas, such as cache:config:1 through cache:config:8, and let the caller pass a random offset when reading. On write, either update every replica (simple and reliable, at the cost of write amplification) or write one and let short TTLs refresh the rest (cheap writes, at the cost of brief inconsistency). For read-heavy, write-rare data, writing all replicas is almost always the better trade.
Finally, nothing gets fixed that is not first identified. OBJECT FREQ with an LFU policy reports per-key access frequency and is the most direct way to find hot keys; in production the OBJECT family can be unavailable behind proxies or restricted permissions, in which case fall back to a short MONITOR sample — never on a peak window — or to client-side instrumentation. Making "top N hot keys this week" a routine report beats debugging by hand at 3 AM.
# Identify hot keys: under an LFU policy use OBJECT FREQ to see access frequency
redis-cli CONFIG SET maxmemory-policy allkeys-lfu
redis-cli --no-raw OBJECT FREQ "hot:config:global"
redis-cli --no-raw SCAN 0 COUNT 500 | head -50 # Roughly sample the key space
# Watch hit rate and evictions
redis-cli --no-raw INFO stats | grep -E "keyspace_hits|keyspace_misses|evicted_keys"
redis-cli --no-raw INFO commandstats | grep -E "cmdstat_(get|set|evict)"
# Capture a slow log sample when troubleshooting (SCAN does not block, but use with care in production)
redis-cli SLOWLOG GET 20An observability checklist before going live
Cache incidents rarely announce themselves as outages; they quietly get slow or quietly get wrong. Instrument three dimensions before launch, not just availability. Hit rate, tracked as keyspace_hits over the sum of hits and misses, should be read as a time series and judged on percentiles rather than averages — a 99 percent average can hide a stretch where it sat at 50 percent. evicted_keys tells you whether the working set genuinely exceeds the memory ceiling, in which case you have merely moved the pressure onto the database.
Collect latency per command: look at the p99 of GET, SET, and EVAL separately. A noticeably higher p99 on EVAL than on ordinary commands usually points to intense lock contention and long lock hold times in the rebuild path. SLOWLOG GET is a tool for post-mortem attribution once something has already gone wrong, not for continuous monitoring.
Prepare a degradation switch. The cache path in code should sit behind a configuration flag so it can be disabled without a code change or a release, degrading to direct database reads: slow, but not a 500. In the same spirit, the fallthrough concurrency ceiling, the single-flight retry limit, and the negative-cache TTL should all be dynamically adjustable configuration rather than hard-coded constants — during an incident you want to tune parameters, not ship a release.
Finally, plan the regression checks. After every change to caching logic, run targeted cases: concurrent load against a hot key, a penetration simulation using random IDs, observation of the expiry-time distribution after a bulk warm-up, and the recovery behaviour after an instance restart. Script these and put them in CI or a scheduled job — their value far exceeds any one-off manual verification.
# Hit rate (sample per time window, do not rely on the cumulative average alone)
redis-cli --no-raw INFO stats | grep -E "keyspace_hits|keyspace_misses"
# Eviction rate: steady growth = the memory limit is no longer enough
redis-cli --no-raw INFO stats | grep evicted_keys
# Latency and command stats
redis-cli --no-raw LATENCY LATEST
redis-cli --no-raw INFO commandstats | grep cmdstat_get
# Memory and fragmentation
redis-cli --no-raw INFO memory | grep -E "used_memory_human|maxmemory_human|allocator_frag_ratio"