Redis caching patterns that cause more downtime than they prevent

5 min read

Most Redis caching patterns that cause more downtime than they prevent share one trait: they push correctness into the cache instead of the database, then silently rot when the cache misbehaves. Teams at Monzo, the BBC and HMRC have all publicly battled Redis incidents where the cache became the single point of failure, not the safety net it was sold as. This post walks through five patterns I keep seeing in UK codebases that turn a fast database into a fragile one, with the real failure modes and what to ship instead.

Cache-as-source-of-truth for writes

The pattern looks innocent: write to Redis first, flush to Postgres asynchronously. It is fast on the happy path and falls apart the moment a Redis node loses its dataset or a pub/sub consumer dies. I have seen this exact shape take down checkout flows because the background flusher crashed and nobody noticed until a customer complained about a duplicate order.

Redis is not a durable store. AOF with appendfsync everysec can lose up to one second of writes; appendfsync no can lose everything since the last OS flush. If your payment service is built on the assumption that Redis has your data, you are one OOM kill away from a P1.

The fix is boring but correct: Postgres (or your system of record) gets the write synchronously, Redis gets a derived view. If you need low write latency, use an outbox table in Postgres and a worker that publishes to Redis. The cache becomes a projection, not the source.

Stale-while-revalidate without locking

This one is endemic in Node and Python services. You serve the cached value, set a short TTL, and rely on the next request to refresh. Under any real traffic shape this is a stampede: when the key expires, hundreds of requests hit Postgres at once. The "fast" cache turns into a thundering herd that takes your primary database offline.

The naïve fix is to bump the TTL. That just delays the stampede and makes the data more stale. Real fix is a single-flight lock so only one process rebuilds the cache while everyone else gets the (slightly) stale value. Here is a working Python sketch using redis-py and SET NX:

import redis, json, time
from myapp.db import get_user

r = redis.Redis()
LOCK_TTL = 5  # seconds

def get_user_cached(user_id):
    key = f"user:{user_id}"
    cached = r.get(key)
    if cached:
        return json.loads(cached)

    lock_key = f"lock:{key}"
    got_lock = r.set(lock_key, "1", nx=True, ex=LOCK_TTL)
    if not got_lock:
        # Someone else is rebuilding; wait briefly then read
        time.sleep(0.05)
        again = r.get(key)
        return json.loads(again) if again else None

    try:
        user = get_user(user_id)  # slow DB call
        r.set(key, json.dumps(user), ex=300)
        return user
    finally:
        r.delete(lock_key)

Notice the lock has a TTL. Without it a crashing worker leaves the key held forever and your cache never refreshes again, which is its own outage. NHS Digital's open write-ups on their appointment booking service show this exact pattern after their 2022 migration.

Key explosion and unbounded namespaces

Developers love putting user IDs, request IDs, search hashes and session tokens into Redis with no TTL and no key prefix strategy. Six months in, the instance is at 92% memory, eviction starts, and suddenly your "cache" is dropping hot keys that were never meant to expire. The latency spikes look random because they are random; eviction is LRU-ish and your most-queried key is not always the most-recently-used.

Set hard rules and enforce them in code review:

• Every SET without EX or PX is a code smell.
• Use a key prefix per domain: user:, session:, rate:. Never let them collide.
• Cap memory with maxmemory and an explicit eviction policy. allkeys-lru is usually wrong for mixed workloads; volatile-lru is safer.
• Export used_memory, used_memory_peak and evicted_keys to Prometheus. If evicted_keys is climbing, your cache is failing open and you are now doing database work in Redis's place.

Read-through caches that hide schema migrations

This one bit a team I worked with at a UK retailer last year. They shipped a read-through cache that deserialised JSON from Redis into a Pydantic model on every request. When they renamed a database column and ran a rolling deploy, half the fleet read the new schema from Postgres and cached it, half the fleet read the old shape and cached it. Cache keys collided on user ID, the two shapes fought for an hour, and the checkout went down.

The mistake was treating the cached JSON as immutable. The fix was a cache key version baked into the key name (user:v2:{id}) and a hard DEL of the old namespace during deploy. Both are ten-minute changes. Doing neither is an outage.

Using Redis for distributed locks on critical paths

Redis-based locks (Redlock, or the popular SET NX pattern) are fine for "do not run this cron twice". They are not fine for "do not double-charge this customer". Clock skew, failover, and partition behaviour mean a Redis lock can be held by two clients at once. Martin Kleppmann's 2016 critique is still the canonical reference, and Antirez's response did not actually close the gap.

If you need mutual exclusion across processes, use Postgres SELECT FOR UPDATE, an advisory lock, or a real consensus system like etcd or ZooKeeper. Redis is great for "probably exclusive, will retry if not". It is bad for "must be exclusive, money on the line".

What actually keeps Redis from taking you down

Three things, in order of importance. First, treat Redis as ephemeral. Anything you cannot reconstruct from Postgres is in the wrong place. Second, instrument it like a database, not a black box: latency_percentiles_usec, instantaneous_ops_per_sec, blocked_clients, rejected_connections, and the slowlog. Third, run a chaos drill. Kill a replica mid-traffic, force a failover, and watch what your service does. The teams that do this find their bad patterns before their customers do.

The common thread across every Redis caching pattern that causes more downtime than it prevents is the same: the cache is being asked to be something it is not. Use it as a derived, lossy, eventually-consistent view of data you already own somewhere durable. The day you need it to be the source of truth is the day you need a different database.

FAQ

What is the most common Redis caching mistake in production?

Treating Redis as a durable store. AOF and RDB are durability hints, not guarantees; under a crash you can lose the last second (or more) of writes. Keep your system of record in Postgres or another ACID database and treat Redis as a derived view.

How do I prevent cache stampedes in Redis?

Use a short-lived lock with SET key value NX EX 5 so only one process rebuilds an expired key while everyone else reads the slightly stale value. Pair it with request coalescing in your application layer if you have heavy fan-out per key.

Should I use Redis for distributed locks on payments or inventory?

No. Redis locks are best-effort and can be violated by failover, partition or clock skew. Use Postgres SELECT FOR UPDATE, advisory locks, or a consensus system like etcd for anything where double-execution causes real damage.